tom@techcolumnist:~$

~/2026/09/09/google-cloud-iam-least-privilege-in-practice-not-just-in-theory.md

Google Cloud: IAM Least Privilege in Practice, Not Just in Theory

---
author: 
date: 
read: 6 min
in:   [engineering]
tags: [gcloud]
---

$ grep -n '^#' post.md

Least privilege is easy to agree with and hard to actually run. Every engineer nods along with the principle right up until their own access gets tightened and something they used to do without thinking now throws a permission error. Getting a Google Cloud organization to genuinely operate on least privilege, not just have it written into a policy document, has been less about picking the right roles and more about building the process that keeps them right after the initial cleanup. This is that process, with the commands and scripts behind each step.

Where least privilege breaks down in practice

The gap almost never shows up at the design stage. It shows up months later, after the original access grants have drifted from whatever was documented, because someone needed a permission urgently, got a broad role to unblock them, and nobody came back to narrow it. Least privilege isn't a state you reach once. It's a maintenance discipline, and the organizations that struggle with it are the ones treating an access review as a one-time project instead of a recurring one.

The first recurring question is always the same: where are the basic roles? Google's own guidance is that basic roles "include thousands of permissions across all Google Cloud services," and that in production you shouldn't grant them unless there's no alternative. Cloud Asset Inventory answers it for the whole organization in one call (you need roles/cloudasset.viewer on the organization):

bash
gcloud asset search-all-iam-policies \
  --scope=organizations/123456789012 \
  --query='roles:roles/owner OR roles:roles/editor' \
  --flatten='policy.bindings[].members[]' \
  --format='table(resource.basename(),policy.bindings.role,policy.bindings.members)'

That search returns bindings attached directly to each resource. It doesn't return access inherited from a folder or the organization. For a full CSV with flags for public, external, and deleted principals, I use the organization-wide IAM audit script. For "who can actually do this one dangerous thing here," inheritance and group membership included, Policy Analyzer is the right tool:

bash
gcloud asset analyze-iam-policy \
  --organization=123456789012 \
  --full-resource-name=//cloudresourcemanager.googleapis.com/projects/payments-prod \
  --permissions='resourcemanager.projects.setIamPolicy' \
  --expand-groups

Budget for that one: more than 20 Policy Analyzer queries per organization per day needs the Premium or Enterprise tier of Security Command Center.

Predefined roles are a starting point, not a destination

Predefined roles are a huge improvement over granting Editor or Owner out of convenience, but most of them are still built for a category of user, not a specific job. A role like roles/compute.admin grants far more than most engineers assigned to it use day to day.

Custom roles solve the precision problem but introduce a maintenance one. Google maintains predefined roles and adds permissions to them as services grow. It doesn't touch your custom roles, so a custom role nobody has updated in two years is quietly falling behind what the underlying service can do. Some permissions also can't go into custom roles at all, and others are only at TESTING support level. Check before you build one:

bash
gcloud iam list-testable-permissions \
  //cloudresourcemanager.googleapis.com/projects/payments-prod \
  --filter='customRolesSupportLevel!=NOT_SUPPORTED AND name:compute.instances.'

When a custom role is worth it, keep it as a file in version control with a named owner, so the review has something to diff:

yaml
# roles/vm-operator.yaml
title: VM Operator
description: Start, stop, reset and view VMs. Owner platform-team@example.com.
stage: GA
includedPermissions:
- compute.instances.get
- compute.instances.list
- compute.instances.reset
- compute.instances.start
- compute.instances.stop
- compute.zoneOperations.get
bash
gcloud iam roles create vmOperator --organization=123456789012 --file=roles/vm-operator.yaml
# later changes
gcloud iam roles update vmOperator --organization=123456789012 --file=roles/vm-operator.yaml

The middle ground we've landed on: predefined roles at the project or folder level for anything broad and low-risk, and custom roles reserved for the handful of high-privilege paths worth the ongoing upkeep. (The hard limit is 3,000 permissions per custom role; if you're anywhere near it, a predefined role was probably the better answer.)

The IAM recommender is only as good as its lookback window

The IAM recommender (google.iam.policy.Recommender) is the fastest way to find over-provisioned bindings. It compares what a principal was granted with what it used over up to 90 days (or since the grant, if that's more recent), and it proposes one of two things: REMOVE_ROLE, or REPLACE_ROLE with something less permissive. There's also REPLACE_ROLE_CUSTOMIZABLE, a suggested custom role. It also uses an ML model to keep permissions a principal is likely to need even if it hasn't used them yet. Recommendations for basic roles are free at the project, folder, and organization level. Recommendations for other predefined roles and custom-role suggestions need Security Command Center Premium or Enterprise. It also doesn't produce insights for conditional bindings.

What it can't tell you is whether an unused permission is unused because it's unnecessary, or because the one task that needs it only runs quarterly and didn't fall inside the window. Accepting every recommendation blind is how you find out, at the worst possible time, that someone's quarter-end batch job needed a permission the recommender flagged as safe to remove. We treat recommendations as a prioritized list to investigate, not a queue to auto-apply.

This script builds that list across every project the caller can see, together with the related list of service accounts that haven't authenticated in 90 days. It needs roles/recommender.iamViewer and roles/iam.roleViewer wherever it runs (granting them at the organization covers every project), plus the Recommender API (recommender.googleapis.com) enabled.

bash
#!/usr/bin/env bash
# iam-review.sh
# Collects active IAM role recommendations and unused-service-account insights
# for every visible project into two CSV files for a periodic access review.
# Usage: ./iam-review.sh [PROJECT_FILTER]
#   PROJECT_FILTER is an optional gcloud --filter, default "lifecycleState=ACTIVE"
set -euo pipefail

FILTER="${1:-lifecycleState=ACTIVE}"
REC_CSV="iam-recommendations-$(date +%F).csv"
SA_CSV="unused-service-accounts-$(date +%F).csv"

echo "project,recommendation_id,subtype,priority,etag,description" > "$REC_CSV"
echo "project,service_account,subtype,severity,description" > "$SA_CSV"

for project in $(gcloud projects list --filter="$FILTER" --format='value(projectId)'); do
  echo "Reviewing $project" >&2

  if ! recs=$(gcloud recommender recommendations list \
      --project="$project" --location=global \
      --recommender=google.iam.policy.Recommender \
      --filter='stateInfo.state=ACTIVE' \
      --format='csv[no-heading](name.basename(),recommenderSubtype,priority,etag,description)' 2>/dev/null); then
    echo "  skipped recommendations (API disabled or no access)" >&2
    recs=""
  fi
  [[ -n "$recs" ]] && sed "s|^|${project},|" <<< "$recs" >> "$REC_CSV"

  if ! sas=$(gcloud recommender insights list \
      --project="$project" --location=global \
      --insight-type=google.iam.serviceAccount.Insight \
      --filter='stateInfo.state=ACTIVE' \
      --format='csv[no-heading](targetResources[0].basename(),insightSubtype,severity,description)' 2>/dev/null); then
    echo "  skipped service account insights" >&2
    sas=""
  fi
  [[ -n "$sas" ]] && sed "s|^|${project},|" <<< "$sas" >> "$SA_CSV"
done

echo "Wrote $REC_CSV ($(($(wc -l < "$REC_CSV") - 1)) rows) and $SA_CSV ($(($(wc -l < "$SA_CSV") - 1)) rows)" >&2

Working a row from that file looks like this. Claiming the recommendation stops the Recommender API from rewriting it while you act, and marking it succeeded closes the loop in the console:

bash
PROJECT=payments-prod
REC_ID=0f1e2d3c-example
ETAG=$(gcloud recommender recommendations describe "$REC_ID" --project="$PROJECT" \
  --location=global --recommender=google.iam.policy.Recommender --format='value(etag)')

gcloud recommender recommendations mark-claimed "$REC_ID" --project="$PROJECT" \
  --location=global --recommender=google.iam.policy.Recommender --etag="$ETAG" \
  --state-metadata=reviewer=platform-team,ticket=SEC-1234

# Apply the change the recommendation describes, for example:
gcloud projects remove-iam-policy-binding "$PROJECT" \
  --member='user:jane@example.com' --role='roles/editor'

ETAG=$(gcloud recommender recommendations describe "$REC_ID" --project="$PROJECT" \
  --location=global --recommender=google.iam.policy.Recommender --format='value(etag)')
gcloud recommender recommendations mark-succeeded "$REC_ID" --project="$PROJECT" \
  --location=global --recommender=google.iam.policy.Recommender --etag="$ETAG"

Applying a recommendation needs roles/recommender.iamAdmin plus the right to change the resource's allow policy. Some project-level recommendations propose a new custom role, which also needs iam.roles.create.

Make broad access temporary instead of permanent

Most "just give me Editor" requests are really "give me more access for this afternoon." IAM conditions make that expire on its own. A binding with a request.time condition stops granting access at the timestamp, and nobody has to remember to remove it:

bash
gcloud projects add-iam-policy-binding payments-prod \
  --member='user:jane@example.com' \
  --role='roles/compute.instanceAdmin.v1' \
  --condition='expression=request.time < timestamp("2026-09-12T00:00:00Z"),title=incident-4711,description=Temporary access for incident 4711'

One limitation pushes you toward predefined roles here anyway: IAM refuses conditions on the basic roles (Owner, Editor, Viewer). For requests that need approval and an audit trail, Privileged Access Manager formalizes the same idea with entitlements that principals request and approvers grant for a bounded duration.

Service accounts are the real risk

Human access gets reviewed because humans are visible: they're in the org chart, they leave the company, someone notices. Service accounts have none of that natural pressure. A service account with a long-lived downloaded key and a broad role is a far bigger blast radius than any single over-permissioned engineer, because it's usually forgotten faster than it's found.

Google's key-management guidance is to use an alternative "whenever possible": an attached service account for workloads on Google Cloud, and Workload Identity Federation for workloads elsewhere (GitHub Actions, AWS, Azure, any OIDC or SAML provider), which authenticates without a downloaded key at all. The organization policy iam.managed.disableServiceAccountKeyCreation enforces that. It's already on by default for organizations created on or after May 3, 2024, and the organization policy post covers rolling it out on older ones.

For the keys that remain, two checks come before any deletion:

bash
# When was each key last used? (roles/policyanalyzer.activityAnalysisViewer)
gcloud policy-intelligence query-activity \
  --activity-type=serviceAccountKeyLastAuthentication \
  --project=payments-prod

# Which service accounts haven't authenticated in 90 days?
gcloud recommender insights list \
  --insight-type=google.iam.serviceAccount.Insight \
  --project=payments-prod --location=global

The service account insight has blind spots Google documents: requests authenticated with API keys bound to a service account aren't counted, and neither is authentication to Google APIs outside Google Cloud, such as Workspace domain-wide delegation. Cross-check with Cloud Monitoring's service account usage metrics before you disable anything. Every remaining downloaded key is a flag for "why does this still need to exist," not a fact of life; the key rotation script is what I use for the ones that really do.

Making it stick

None of this holds without the boring part: a recurring review cadence (the two CSVs above, every month, diffed against last month), an owner for every custom role, and enough cultural buy-in that a tightened permission gets reported as friction to investigate rather than quietly worked around with a personal access grant. The technical controls are the easy half. The discipline to keep re-applying them as the organization changes is the part that actually determines whether least privilege is real or just written down.

References