OpenStack Multi-Tenancy Architecture: Glance, Keystone, and Cross-Tenant Resource Sharing

December 22, 2020

Multi-tenancy is one of OpenStack's fundamental design principles, enabling secure resource isolation while allowing controlled sharing when needed. Understanding how this works at the architectural level, particularly the interaction between Keystone's authentication layer and Glance's image management is crucial for operating production OpenStack deployments.

OpenStack Multi-Tenancy Architecture

Keystone: Projects, Domains, and RBAC

Keystone implements multi-tenancy through a hierarchical model:

# tree
Domain
  └── Projects (Tenants)
       └── Users + Roles

Projects (formerly called tenants) are the fundamental isolation boundary. Each project has its own quota, resource ownership, and access control. Domains provide an additional layer of organization, allowing multiple projects to be grouped, useful for large organizations with multiple departments.

Role-Based Access Control (RBAC) determines what users can do within a project. Common roles include:

admin: Full control within a project
member: Standard user operations
reader: Read-only access

These roles are enforced through policy.json files in each OpenStack service.

Token Scoping Mechanisms

Keystone tokens can be scoped in different ways, fundamentally changing what resources they can access:

# Unscoped token - can list projects but access no resources
openstack token issue

# Project-scoped token - access resources in specific project
openstack --os-project-name production token issue

# Domain-scoped token - administrative operations across projects
openstack --os-domain-name engineering token issue

When a user authenticates, Keystone returns a token containing:

User identity and roles
Scope (project/domain)
Service catalog (API endpoints)
Expiration timestamp

This token is presented to every OpenStack service (Nova, Glance, Cinder, etc.), which validates it against Keystone and enforces authorization based on the scope and roles.

Control Plane vs Data Plane Authorization

OpenStack separates two authorization layers:

Control Plane: API-level authorization enforced by policy.json rules. Example from Glance:

{
  "get_image": "role:admin or role:member",
  "add_member": "role:admin",
  "modify_member": "role:admin"
}

Data Plane: Database-level ownership and visibility. Even if you have API access, you can only see resources you own or that have been explicitly shared with you.

Glance Image Management Architecture

Glance is the OpenStack Image Service, responsible for discovering, registering, and delivering virtual machine images. Its architecture cleanly separates concerns.

Image Registry vs Image Store

Glance uses a two-component architecture:

glance-registry: Manages image metadata in a database (MySQL). Stores:

Image properties (name, size, format, container format)
Ownership (project_id)
Visibility (public/private/shared/community)
Custom metadata (key-value pairs)
Image members (sharing relationships)

glance-api: Handles API requests and interfaces with storage backends. Acts as the gateway between clients and actual image data.

This separation allows metadata operations (listing images, updating properties) to be fast database queries, while image data transfers happen through optimized storage backends.

Backend Storage Drivers

Glance supports multiple storage backends, each with different characteristics:

# In /etc/glance/glance-api.conf

[glance_store]
stores = file,http,swift,rbd
default_store = rbd

[glance_store:rbd]
rbd_store_pool = images
rbd_store_user = glance
rbd_store_ceph_conf = /etc/ceph/ceph.conf

Storage backend comparison:

Ceph (RBD): High performance, distributed, supports CoW (copy-on-write) for fast cloning
Swift: Object storage, highly scalable, geographically distributed
Filesystem: Simple, limited scalability, useful for small deployments
HTTP: Read-only store for importing images from HTTP/HTTPS URLs

Image Metadata Schema

Every image has core metadata and supports custom properties:

$ openstack image create \
  --disk-format qcow2 \
  --container-format bare \
  --min-disk 10 \
  --min-ram 1024 \
  --property os_type=linux \
  --property os_distro=ubuntu \
  --property os_version=20.04 \
  --file ubuntu-20.04.qcow2 \
  ubuntu-base

+------------------+------------------------------------------------------+
| Field            | Value                                                |
+------------------+------------------------------------------------------+
| checksum         | 4d8b3f50b6e7c8a2d9f1e3a5c6b8d9e1                     |
| container_format | bare                                                 |
| created_at       | 2020-12-15T10:30:00Z                                 |
| disk_format      | qcow2                                                |
| file             | /v2/images/a3c7b4d1-2e8f-4a9b-b2c1-5d6e7f8a9b0c/file |
| id               | a3c7b4d1-2e8f-4a9b-b2c1-5d6e7f8a9b0c                 |
| min_disk         | 10                                                   |
| min_ram          | 1024                                                 |
| name             | ubuntu-base                                          |
| os_distro        | ubuntu                                               |
| os_type          | linux                                                |
| os_version       | 20.04                                                |
| owner            | f4e8d2c1a3b5e7d9c8b6a4f2e1d3c5b7                     |
| protected        | False                                                |
| schema           | /v2/schemas/image                                    |
| size             | 2361393152                                           |
| status           | active                                               |
| tags             |                                                      |
| updated_at       | 2020-12-15T10:31:45Z                                 |
| virtual_size     | None                                                 |
| visibility       | private                                              |
+------------------+------------------------------------------------------+

Custom properties enable advanced functionality:

hw_scsi_model=virtio-scsi: Use virtio SCSI for better performance
hw_disk_bus=scsi: Set disk bus type
os_secure_boot=required: Enable UEFI Secure Boot
img_signature: GPG signature for image verification

Image Visibility States

Glance defines four visibility levels:

public: Visible to all projects (requires admin to create)
private: Visible only to owning project (default)
shared: Shared with specific projects via image members
community: Shared with all, but not in default listings

The state machine:

private → shared → community
   │         ↑              ↑
   └─────────┴──────────────┘
          (admin only)

Image Membership Model

The glance.image_members table implements sharing (see Glance API v2 Image Members implementation):

CREATE TABLE image_members (
  id INT PRIMARY KEY AUTO_INCREMENT,
  image_id VARCHAR(36) NOT NULL,
  member VARCHAR(255) NOT NULL,  -- target project_id
  can_share BOOLEAN DEFAULT FALSE,
  status ENUM('pending', 'accepted', 'rejected') DEFAULT 'pending',
  created_at DATETIME,
  updated_at DATETIME,
  deleted_at DATETIME,
  deleted BOOLEAN DEFAULT FALSE,
  UNIQUE KEY (image_id, member)
);

The status field implements a three-way handshake for sharing, preventing spam.

Image Lifecycle State Machine

Images transition through several states:

queued → saving → active → deactivated → deleted
                    ↓
                 pending_delete

queued: Metadata created, no data uploaded yet
saving: Data upload in progress
active: Ready for use
deactivated: Temporarily disabled (admin action)
pending_delete: Scheduled for removal
deleted: Soft-deleted (recoverable)

Cross-Tenant Image Sharing

The complete workflow for sharing an image between projects involves several steps and database changes.

The Complete API Workflow

Step 1: Image owner creates member

# As project A (owner), share image with project B
$ openstack image add project \
  my-ubuntu-image \
  <project-b-id>

# Low-level API call
$ curl -X POST \
  -H "X-Auth-Token: $TOKEN_PROJECT_A" \
  -H "Content-Type: application/json" \
  http://glance:9292/v2/images/<image-id>/members \
  -d '{"member": "<project-b-id>"}'

The query that happens in database:

INSERT INTO image_members
(image_id, member, status, created_at)
VALUES
('<image-id>', '<project-b-id>', 'pending', NOW());

At this point, project B cannot see the image yet.

Step 2: Project B accepts the share

# As project B, list pending shares
$ openstack --os-project-name project-b image member list <image-id>

# Accept the share
$ openstack --os-project-name project-b image set \
  --accept \
  <image-id>

# Low-level API call
$ curl -X PUT \
  -H "X-Auth-Token: $TOKEN_PROJECT_B" \
  -H "Content-Type: application/json" \
  http://glance:9292/v2/images/<image-id>/members/<project-b-id> \
  -d '{"status": "accepted"}'

The query that happens in database:

UPDATE image_members
SET status = 'accepted', updated_at = NOW()
WHERE image_id = '<image-id>' AND member = '<project-b-id>';

Step 3: Verification

# Project B can now see and use the image
$ openstack --os-project-name project-b image list

$ openstack --os-project-name project-b server create \
  --image my-ubuntu-image \
  --flavor m1.small \
  test-instance

Policy.json Rules

The sharing workflow is governed by policies in /etc/glance/policy.json:

{
  "add_member": "role:admin or (role:member and project_id:%(owner)s)",
  "get_member": "role:admin or project_id:%(owner)s or project_id:%(member)s",
  "modify_member": "role:admin or (role:member and project_id:%(member)s)",
  "delete_member": "role:admin or (role:member and project_id:%(owner)s)"
}

Key points:

Only the image owner can add members
Only the target project can accept/reject membership
Both owner and member can view membership status
Only owner can revoke sharing

Keystone Token Validation Integration

When project B makes an API call to Glance:

Glance receives the request with project B's token
Glance validates token with Keystone (cached for performance)
Glance extracts project_id from token context

Glance queries database:

SELECT * FROM images i
LEFT JOIN image_members m ON i.id = m.image_id
WHERE i.owner = '<project-b-id>'
   OR i.visibility = 'public'
   OR (m.member = '<project-b-id>' AND m.status = 'accepted');

Policy.json rules are evaluated with context variables
If authorized, the operation proceeds

Sharing vs Cloning

Two approaches to make an image available across projects:

Image Sharing (membership):

Single image, multiple viewers
No storage duplication
Owner controls lifecycle (can delete/modify)
Changes propagate to all members
Uses image_members table

Image Cloning:

# Download from project A
openstack image save --file /tmp/image.qcow2 <image-id>

# Upload to project B
openstack --os-project-name project-b image create \
  --file /tmp/image.qcow2 \
  --disk-format qcow2 \
  cloned-image

Complete copy, independent lifecycle
Double storage consumption
Full ownership by target project
Immune to source image changes
Better for long-term independence

With Ceph RBD backend, cloning is more efficient:

# Create copy-on-write clone (instant, no data copy)
rbd clone images/<image-id>@snap images/<new-image-id>

Architectural Tradeoffs

Sharing vs Cloning: When to Use Each

Use sharing when:

Image is large (100GB+ VM templates)
Centralized image management desired
Regular updates need to propagate
Storage efficiency is critical
Short-term collaboration

Use cloning when:

Projects need full control
Image stability required (prod vs dev)
Compliance requires data isolation
Projects might outlive image owner
Network transfer is acceptable

Security Boundaries and Tenant Isolation

OpenStack provides strong isolation guarantees:

Network isolation: Neutron ensures projects cannot see each other's traffic
Storage isolation: Ceph uses project-specific pools/namespaces
Compute isolation: Hypervisor (KVM/QEMU) provides process isolation
API isolation: All cross-tenant operations require explicit authorization

However, shared images introduce a trust boundary:

Malicious image owner could include backdoors
Image updates affect all consumers
Metadata visibility (image size, format) is shared

Best practices:

Use image signature verification
Clone images for production workloads
Implement image scanning pipelines
Restrict public image creation to admins

Advanced Topics

Image Deactivation and Reactivation

Admins can temporarily disable images without deletion:

# Deactivate image (prevents new instance creation)
openstack image set --deactivate <image-id>

# Existing instances continue running
# New instance creation fails:
# ERROR: Image <image-id> is deactivated

# Reactivate when issue resolved
openstack image set --activate <image-id>

Database transition:

UPDATE images SET status = 'deactivated' WHERE id = '<image-id>';
-- Later...
UPDATE images SET status = 'active' WHERE id = '<image-id>';

Use cases:

Security vulnerability discovered
Licensing issues
Corrupted image data
Maintenance windows

Multi-Backend Configuration

Glance can use different backends for different images:

[glance_store]
stores = ceph_ssd,ceph_hdd,swift_backup
default_store = ceph_ssd

[glance_store:ceph_ssd]
rbd_store_pool = images-fast

[glance_store:ceph_hdd]
rbd_store_pool = images-archive

[glance_store:swift_backup]
swift_store_auth_address = https://swift.example.com/auth/v1.0

Migrating images between backends:

# Copy image to different backend
glance --os-image-api-version 2 image-import <image-id> \
  --method copy-image \
  --store ceph_hdd

Image Import Methods

Glance v2 supports three import methods:

1. glance-direct (default): Upload directly to Glance

openstack image create --file image.qcow2 my-image

2. web-download: Glance downloads from URL

openstack image create my-image
openstack image set my-image --property hw_disk_bus=scsi

# Trigger import
curl -X POST \
  -H "X-Auth-Token: $TOKEN" \
  -H "Content-Type: application/json" \
  http://glance:9292/v2/images/<image-id>/import \
  -d '{
    "method": {
      "name": "web-download",
      "uri": "https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img"
    }
  }'

3. copy-image: Copy between Glance stores

glance image-import <image-id> --method copy-image --store swift_backup

Image Signature Verification

For trusted image pipelines:

# Generate GPG signature
gpg --detach-sign --armor image.qcow2

# Upload with signature
openstack image create \
  --file image.qcow2 \
  --property img_signature=$(cat image.qcow2.asc | base64 -w0) \
  --property img_signature_hash_method=SHA-256 \
  --property img_signature_key_type=GPG \
  --property img_signature_certificate_uuid=$KEY_UUID \
  trusted-ubuntu

Nova can be configured to reject unsigned images:

[glance]
verify_glance_signatures = true

Glance Integration with Cinder and Nova

Cinder (volumes from images):

# Create bootable volume from image
openstack volume create \
  --image ubuntu-20.04 \
  --size 20 \
  --bootable \
  my-boot-volume

Backend flow:

Cinder calls Glance API to get image location
Cinder uses qemu-img convert or Ceph RBD clone
Volume marked as bootable with image metadata

Nova (ephemeral disk creation):

openstack server create \
  --image ubuntu-20.04 \
  --flavor m1.small \
  instance-1

Backend flow:

Nova queries Glance for image details
Nova creates ephemeral disk (Ceph RBD clone or qcow2 backing file)
Nova boots VM with disk attached

With Ceph RBD:

# Nova creates instant CoW clone
rbd clone images/ubuntu-20.04@snap vms/instance-1_disk

Ceph RBD Cloning: Flatten vs Not-Flatten

When Cinder creates volumes from Glance images stored in Ceph RBD, it can either create a copy-on-write (CoW) clone or flatten the volume. This choice has significant performance implications for production clusters.

The Clone Hierarchy Problem:

Glance Image (parent)
    └── Volume Clone (child)
         └── Snapshot
              └── Another Clone (grandchild)

Each clone maintains a dependency on its parent, creating a chain that Ceph must traverse on every read operation that hasn't been overwritten yet.

Performance Impact Analysis

Flattening (decoupling from parent):

When you flatten a volume, Ceph copies all data from the parent image into the child volume, making it independent:

# Flatten a volume manually
rbd flatten volumes/volume-abc123

Impact on OSDs during flattening:

Massive sequential I/O: The entire parent image is copied to the volume
OSD saturation: Sequential writes can consume hundreds of MB/s per OSD
Immediate Slow Ops: Other VMs experience degraded performance during the operation
Duration: Can take 10+ minutes for a 100GB volume depending on cluster load

Not Flattening (keeping CoW clone):

Volumes remain as thin clones, sharing data with the parent image:

# Check clone relationship
rbd info volumes/volume-abc123
# Shows: parent: images/ubuntu-20.04@snap

Impact on ongoing operations:

Random I/O spikes: Every first read of a block triggers a "copy-up" operation
Increased metadata lookups: Ceph must check parent chain for unchanged blocks
Chronic Slow Ops: Sustained performance degradation during normal VM operations
Higher read latency: 2-10x increase for reads hitting parent blocks
Deep clone chains: >5 levels can severely impact performance

Cinder Configuration Options

In /etc/cinder/cinder.conf:

[DEFAULT]
# Clone from Glance image (fast initial creation)
rbd_flatten_volume_from_snapshot = false

# Enable copy-on-read to gradually flatten on-demand
rbd_clone_copy_on_read = true

# Maximum clone depth before forcing flatten
rbd_max_clone_depth = 5

Blueprint reference: Cinder Flatten Volume from Image

Real-World Impact Summary

Scenario	Impact on OSDs	Consequence
Flattening	Massive, sustained sequential I/O while decoupling	Immediate Slow Ops due to OSD saturation during the move
Not Flattening	Random I/O spikes (copy-ups) and increased metadata lookups	Chronic Slow Ops during normal VM operations and high read latency

Production Recommendations

1. Use rbd_clone_copy_on_read = true:

This enables Copy-on-Read, which moves data to the clone upon first read, effectively "flattening" the image on-demand:

[DEFAULT]
rbd_clone_copy_on_read = true

Benefits:

Gradual flattening during normal operations
No massive I/O spike
Parent blocks copied only when accessed
Eventually achieves independent volume

2. Avoid Long Clone Chains:

Limit clone depth to prevent performance degradation:

[DEFAULT]
rbd_max_clone_depth = 5

If your Glance configuration allows deep hierarchies (e.g., snapshot → clone → snapshot → clone), ensure you're not exceeding 5-10 levels. Each level adds latency to read operations.

3. Flatten During Low Traffic:

If you must flatten to decouple from a parent (e.g., before deleting a "golden image"), schedule it during off-peak hours:

# Flatten all volumes derived from an image before deletion
for vol in $(rbd ls volumes | grep parent-image); do
    rbd flatten volumes/$vol &
done
wait

This avoids competing with production VM traffic and prevents widespread Slow Ops warnings.

4. Monitor Clone Depth:

Track clone chain depth in your monitoring:

# Check clone depth for a volume
rbd info volumes/volume-abc123 | grep parent

# Recursive script to find depth
function clone_depth() {
    local img=$1
    local depth=0
    while rbd info $img 2>/dev/null | grep -q parent; do
        img=$(rbd info $img | grep parent | awk '{print $2}')
        ((depth++))
    done
    echo $depth
}

clone_depth volumes/volume-abc123

Alert when depth exceeds 5 to trigger manual flattening or investigation.

5. Pre-flatten Critical Workload Volumes:

For performance-sensitive workloads (databases, high-IOPS applications), flatten volumes immediately after creation:

# Create volume from image
openstack volume create --image ubuntu-db --size 100 db-volume

# Get volume ID and flatten
VOL_ID=$(openstack volume show db-volume -f value -c id)
rbd flatten volumes/volume-$VOL_ID

This ensures predictable, low-latency I/O without parent lookups.

Debugging Clone-Related Performance Issues

Symptom: VMs experience high I/O latency and Ceph shows "slow ops"

# Check for slow ops in Ceph
ceph health detail
# HEALTH_WARN 15 slow ops, oldest one blocked for 32 sec

# Identify which volumes have deep clone chains
for vol in $(rbd ls volumes); do
    depth=$(clone_depth volumes/$vol)
    if [ $depth -gt 3 ]; then
        echo "Volume $vol has depth $depth"
        rbd info volumes/$vol
    fi
done

Solution: Flatten volumes with deep chains during maintenance window or enable rbd_clone_copy_on_read to gradually resolve the issue.

Technical Troubleshooting

Keystone Role Assignment Issues

Issue: Member acceptance fails with 403 Forbidden.

$ glance member-update <image-id> <project-id> accepted
403 Forbidden: Policy doesn't allow modify_member to be performed

Diagnosis: Verify user has correct role in target project:

openstack role assignment list \
  --user $USER_ID \
  --project $PROJECT_ID \
  --names

Fix:

openstack role add \
  --user john.doe \
  --project project-b \
  member

Image Member State Machine Errors

Issue: Image shows as shared but isn't visible in target project.

Diagnosis: Check member status in database:

SELECT
  i.name,
  m.member AS project_id,
  m.status,
  m.created_at
FROM images i
JOIN image_members m ON i.id = m.image_id
WHERE i.id = '<image-id>';

Expected output:

+------------+------------------+----------+---------------------+
| name       | project_id       | status   | created_at          |
+------------+------------------+----------+---------------------+
| my-ubuntu  | abc123...        | accepted | 2020-01-22 10:30:00 |
+------------+------------------+----------+---------------------+

If status = 'pending', the target project needs to accept:

glance member-update <image-id> <project-id> accepted

Storage Backend Connectivity Problems

Issue: Image uploads hang or fail with timeout.

Diagnosis: Test Ceph connectivity:

# Check Ceph cluster health
ceph health detail

# Test RBD access with Glance credentials
rbd --id glance --keyring /etc/ceph/ceph.client.glance.keyring \
  ls images

# Monitor Glance API logs
sudo tail -f /var/log/glance/glance-api.log | grep -i error

Common errors:

ERROR glance.common.client [req-...] Connection timeout to Ceph cluster

Fix: Verify network connectivity and credentials:

# Test network to Ceph monitors
telnet ceph-mon-1 6789

# Verify keyring permissions
ls -la /etc/ceph/ceph.client.glance.keyring
# Should be readable by glance user

Debugging with Glance Logs and SQL Queries

Enable debug logging in /etc/glance/glance-api.conf:

[DEFAULT]
debug = True
log_file = /var/log/glance/glance-api.log

Monitor API requests:

sudo tail -f /var/log/glance/glance-api.log | \
  grep -E "(POST|PUT|DELETE|ERROR)"

Useful database queries:

-- Find all images shared with project
SELECT
  i.id,
  i.name,
  i.owner,
  i.visibility,
  m.status AS share_status
FROM images i
LEFT JOIN image_members m ON i.id = m.image_id
WHERE m.member = '<project-id>' OR i.owner = '<project-id>'
ORDER BY i.created_at DESC;

-- Find orphaned image members (target project deleted)
SELECT m.* FROM image_members m
LEFT JOIN keystone.project p ON m.member = p.id
WHERE p.id IS NULL AND m.deleted = 0;

-- Check image storage usage by project
SELECT
  owner AS project_id,
  COUNT(*) AS image_count,
  SUM(size) / 1024 / 1024 / 1024 AS total_gb
FROM images
WHERE deleted = 0
GROUP BY owner
ORDER BY total_gb DESC;

Conclusion

OpenStack's multi-tenancy architecture demonstrates sophisticated separation of concerns: Keystone handles authentication and authorization, Glance manages image lifecycle and metadata, and the cross-tenant sharing mechanism provides controlled collaboration without sacrificing isolation.

Understanding these internals, from token scoping to database schema to policy enforcement, enables operators to troubleshoot issues quickly, implement custom workflows, and optimize for specific use cases. The key insight is that multi-tenancy isn't just API-level filtering; it's a comprehensive system spanning authentication tokens, database constraints, policy rules, and storage backends working together to provide both isolation and controlled sharing.

For production deployments, remember:

Use image signature verification for security
Clone images for critical workloads to avoid dependency on other projects
Monitor quota usage and enforce limits at both API and database layers
Implement proper RBAC through Keystone and policy.json
Choose storage backends based on your performance and scale requirements
Maintain detailed logs for audit and troubleshooting

The multi-tenancy model is what makes OpenStack suitable for large-scale cloud deployments, enabling thousands of projects to safely coexist while sharing infrastructure efficiently.

References

Special thanks to my mentor whose blog posts on OpenStack provided foundational knowledge for this deep dive:

Layanan Keystone untuk Autentikasi Identitas di OpenStack (Bagian 2) - Comprehensive guide to Keystone authentication and identity services
OpenStack Image Service: Managing Snapshots & Backups - Detailed coverage of Glance image management
OpenStack Glance Official Documentation - Glance architecture and API
Cinder Flatten Volume from Image Blueprint - Specification for Ceph RBD volume flattening feature