extensions: add sync, download functionalities #11814

shwstppr · 2025-10-09T09:40:16Z

Description

This change introduces support for:

synchronizing extension files across management servers in a clustered environment.
download extension files as an archive (from a desired management server in a clustered environment). If needed (cases when MS is not directly accessible), download can be made to work via SSVM and secondary storage.

Key changes:

Adds new syncExtension API to trigger synchronization for a selected extension.
Implements util to create .tgz archive of the extension directory or selected files for sync.
For sync, sends DownloadAndSyncExtensionFilesCommand to peer management servers. Handles archive download, staging, and extraction on the receiver side.
Adds new downloadExtension API to allow downloading extension files as an archive.
Implements util to create .zip archive of the extension directory for download.
For forcing download via secondary storage, global configuration - extension.share.download.use.secondary.storage can be used and set to true.
Generates signed share URL with HMAC signature and expiry.
URL expiry and file cleanup can be controlled via global configuration - extension.share.link.validity.interval
Adds Sync Extension and Download Extension actions in the UI Extensions view.
Updates events, logging, and cleanup of temporary share files.
Refactors all filesystem-related code to the extensions framework layer.
Checksum for extensions is now calculated and compared for all files in the extension directory.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

For Sync:

(localcloud) 🐱 > sync extension id=3cf82f18-89c5-4acc-8f46-8d42da139f30 sourcemanagementserverid=0d8d73fa-9241-49aa-8d7e-7642f143ce66
{
  "success": true
}
(localcloud) 🐱 > sync extension id=3cf82f18-89c5-4acc-8f46-8d42da139f30
💩 Missing required parameters:  sourcemanagementserverid
(localcloud) 🐱 > sync extension -h
syncExtension: To sync the extension files from one management server to other management server(s)
This API is asynchronous.
Required params: id, sourcemanagementserverid, 
API Params               Type     Description
==========               ====     ===========
files                    list     List of files to sync. Specify absolute o
                                  r relative paths. If not provided, all e
                                  xtension files will be synced.
id                       uuid     ID of the extension
sourcemanagementserverid uuid     ID of the management server from which fi
                                  les are to be synced
targetmanagementserverids list     the IDs of the management servers to whic
                                  h the extension files are to be synced. 
                                  If not specified, the files will be sync
                                  ed to all management servers

For Download:

(localcloud) 🐱 > download extension -h
downloadExtension: To download the extension files as an archive
This API is asynchronous.
Required params: id, 
API Params               Type     Description
==========               ====     ===========
id                       uuid     ID of the extension
managementserverid       uuid     ID of the management server from which fi
                                  les are to be downloaded
(localcloud) 🐱 > download extension id=3cf82f18-89c5-4acc-8f46-8d42da139f30
{
  "extension": {
    "id": "3cf82f18-89c5-4acc-8f46-8d42da139f30",
    "managementserverid": "0d8d73fa-9241-49aa-8d7e-7642f143ce66",
    "managementservername": "qa1-4.22-test-61f54c88-kvm-mgmt1",
    "name": "test",
    "url": "http://172.120.0.228:8080/share/extensions/test-1760334975686.zip?exp=1760338575"
  }
}
(localcloud) 🐱 > download extension id=3cf82f18-89c5-4acc-8f46-8d42da139f30 managementserverid=5ccb810f-4a31-4b8d-a36c-2ebb7e87cb26 
{
  "extension": {
    "id": "3cf82f18-89c5-4acc-8f46-8d42da139f30",
    "managementserverid": "5ccb810f-4a31-4b8d-a36c-2ebb7e87cb26",
    "managementservername": "422-test-mgmt2",
    "name": "test",
    "url": "http://172.120.0.83:8080/share/extensions/test-1760334994612.zip?exp=1760338594"
  }
}

How Has This Been Tested?

How did you try to break this feature and the system with this change?

codecov · 2025-10-09T09:48:28Z

Codecov Report

❌ Patch coverage is 61.69031% with 621 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.02%. Comparing base (a5b6bc3) to head (21419d1).

Files with missing lines	Patch %	Lines
...extensions/manager/ExtensionsShareManagerImpl.java	76.47%	66 Missing and 50 partials ⚠️
...sions/manager/ExtensionsFilesystemManagerImpl.java	65.35%	64 Missing and 33 partials ⚠️
...work/extensions/manager/ExtensionsManagerImpl.java	73.14%	24 Missing and 34 partials ⚠️
...amework/extensions/ExtensionArchiveDataObject.java	12.50%	48 Missing and 1 partial ⚠️
...dstack/storage/to/DownloadableArchiveObjectTO.java	0.00%	48 Missing ⚠️
.../cloudstack/utils/server/ServerPropertiesUtil.java	0.00%	39 Missing ⚠️
...om/cloud/storage/download/DownloadMonitorImpl.java	0.00%	31 Missing ⚠️
...oudstack/storage/template/DownloadManagerImpl.java	0.00%	31 Missing ⚠️
...dstack/storage/image/BaseImageStoreDriverImpl.java	0.00%	28 Missing ⚠️
...pache/cloudstack/utils/filesystem/ArchiveUtil.java	69.04%	20 Missing and 6 partials ⚠️
... and 17 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11814      +/-   ##
============================================
+ Coverage     17.89%   18.02%   +0.12%     
- Complexity    16092    16313     +221     
============================================
  Files          5936     5951      +15     
  Lines        532734   534054    +1320     
  Branches      65165    65307     +142     
============================================
+ Hits          95347    96263     +916     
- Misses       426711   426984     +273     
- Partials      10676    10807     +131

Flag	Coverage Δ
uitests	`3.67% <ø> (-0.01%)`	⬇️
unittests	`19.13% <61.69%> (+0.13%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This change introduces support for synchronizing extension files across management servers in a clustered environment. - Adds new syncExtension API to trigger synchronization for a selected extension. - Implements service to create .tgz archive of the extension directory or selected files. - Generates signed share URL with HMAC signature and expiry. - Sends DownloadAndSyncExtensionFilesCommand to peer management servers. - Handles archive download, staging, and extraction on receiver side. - Adds Sync Extension action in the UI Extensions view. - Updates events, logging, and cleanup of temporary share files. Refactors all filesystem related code to extensions framework layer. Checksum for extensions is now calculated and compared for all files in the extension directory. Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

github-actions · 2025-10-17T13:28:55Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

github-actions · 2025-12-26T07:04:45Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr · 2025-12-30T06:36:42Z

@blueorangutan package

blueorangutan · 2025-12-30T06:38:11Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-12-30T08:21:31Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16208

github-actions · 2026-01-27T15:28:47Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

borisstoyanov · 2026-01-30T07:16:12Z

@blueorangutan package

blueorangutan · 2026-01-30T07:18:04Z

@borisstoyanov a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-01-30T08:12:44Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16643

harikrishna-patnala

@shwstppr this looks great. Got a chance to test, sync and downloads are working fine. Though link expiry does not seem to be working as expected.
I've enabled this "extension.share.download.use.secondary.storage" and set "extension.share.link.validity.interval" to 30.

I've made 3 attempts of download, folders and the last file remains exists after expiry time

github-actions · 2026-01-30T15:58:44Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

shwstppr · 2026-01-31T04:39:07Z

@harikrishna-patnala thanks a lot for review/testing. Will have a look at the cleanup flow.

url Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr · 2026-01-31T08:54:11Z

@shwstppr this looks great. Got a chance to test, sync and downloads are working fine. Though link expiry does not seem to be working as expected. I've enabled this "extension.share.download.use.secondary.storage" and set "extension.share.link.validity.interval" to 30.

I've made 3 attempts of download, folders and the last file remains exists after expiry time

@harikrishna-patnala this is a generic issue with download url cleanup on SSVM. While the symlink was removed, temp directory was not getting removed.

2026-01-31T07:36:09,043 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-5:[]) Request:Seq 4-7227995926952935435:  { Cmd , MgmtId: 90520731363323, via: 4, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.storage.DeleteEntityDownloadURLCommand":{"path":"volumes/2/4/97b241b4-4d31-469f-b4a4-a2fcab2606ff.qcow2","extractUrl":"http://172.120.1.2/userdata/88a99588-c0e0-4691-96c7-c7e6534d9fbe/t-dd.qcow2","type":"VOLUME","parentPath":"c794fe3b-bedb-3e8e-a5c6-c5a2b28a2065","accountId":"0","wait":"0","bypassHostMaintenance":"false"}}] }
2026-01-31T07:36:09,044 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-5:[]) Processing command: com.cloud.agent.api.storage.DeleteEntityDownloadURLCommand
2026-01-31T07:36:09,044 DEBUG [storage.resource.NfsSecondaryStorageResource] (AgentRequest-Handler-5:[]) Executing command "DeleteEntityDownloadURLCommand" [{"path":"volumes/2/4/97b241b4-4d31-469f-b4a4-a2fcab2606ff.qcow2","extractUrl":"http://172.120.1.2/userdata/88a99588-c0e0-4691-96c7-c7e6534d9fbe/t-dd.qcow2","type":"VOLUME","parentPath":"c794fe3b-bedb-3e8e-a5c6-c5a2b28a2065","accountId":0,"contextMap":{},"wait":0,"bypassHostMaintenance":false}].
2026-01-31T07:36:09,044 WARN  [storage.template.UploadManagerImpl] (AgentRequest-Handler-5:[]) handleDeleteEntityDownloadURLCommand Path:volumes/2/4/97b241b4-4d31-469f-b4a4-a2fcab2606ff.qcow2 Type:VOLUME
2026-01-31T07:36:09,045 DEBUG [storage.template.UploadManagerImpl] (AgentRequest-Handler-5:[]) Executing command [/bin/bash -c unlink /var/www/html/userdata/88a99588-c0e0-4691-96c7-c7e6534d9fbe/t-dd.qcow2 ].
2026-01-31T07:36:09,054 DEBUG [storage.template.UploadManagerImpl] (AgentRequest-Handler-5:[]) Successfully executed process [5938] for command [/bin/bash -c unlink /var/www/html/userdata/88a99588-c0e0-4691-96c7-c7e6534d9fbe/t-dd.qcow2 ].
2026-01-31T07:36:09,054 WARN  [storage.template.UploadManagerImpl] (AgentRequest-Handler-5:[]) Deleted volume /mnt/SecStorage/c794fe3b-bedb-3e8e-a5c6-c5a2b28a2065/volumes/2/4/97b241b4-4d31-469f-b4a4-a2fcab2606ff.qcow2
2026-01-31T07:36:09,055 DEBUG [storage.template.UploadManagerImpl] (AgentRequest-Handler-5:[]) Executing command [/bin/bash -c rm -rf /mnt/SecStorage/c794fe3b-bedb-3e8e-a5c6-c5a2b28a2065/volumes/2/4/97b241b4-4d31-469f-b4a4-a2fcab2606ff.qcow2 ].
2026-01-31T07:36:09,063 DEBUG [storage.template.UploadManagerImpl] (AgentRequest-Handler-5:[]) Successfully executed process [5939] for command [/bin/bash -c rm -rf /mnt/SecStorage/c794fe3b-bedb-3e8e-a5c6-c5a2b28a2065/volumes/2/4/97b241b4-4d31-469f-b4a4-a2fcab2606ff.qcow2 ].
2026-01-31T07:36:09,063 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-5:[]) Seq 4-7227995926952935435:  { Ans: , MgmtId: 90520731363323, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":"true","details":"","wait":"0","bypassHostMaintenance":"false"}}]


root@s-2-VM:/var/www/html/userdata# ls -lrt
total 20
drwxr-xr-x 2 www-data www-data 4096 Jan 31 06:08 9142b6e3-f6c6-48de-86b6-6003eb147d2c
drwxr-xr-x 2 www-data www-data 4096 Jan 31 06:10 85378760-8b68-4c60-81fc-11ddbe59fd85
drwxr-xr-x 2 www-data www-data 4096 Jan 31 06:28 a3276a6b-03aa-44d0-aeef-183da3b214c4
drwxr-xr-x 2 www-data www-data 4096 Jan 31 06:35 0cfcf0a0-58f8-471b-8864-9b8a2b754da3
drwxr-xr-x 2 www-data www-data 4096 Jan 31 07:36 88a99588-c0e0-4691-96c7-c7e6534d9fbe
root@s-2-VM:/var/www/html/userdata# ls -lrt 88a99588-c0e0-4691-96c7-c7e6534d9fbe
total 0

I've added a fix and created #12562

shwstppr · 2026-01-31T08:54:27Z

@blueorangutan package

blueorangutan · 2026-01-31T08:56:04Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-01-31T09:38:48Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16662

shwstppr · 2026-01-31T19:10:48Z

@blueorangutan test

blueorangutan · 2026-01-31T19:12:03Z

@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

boring-cyborg bot added component:api component:management-server component:UI labels Oct 9, 2025

shwstppr force-pushed the feature-sync-ext branch from af72efe to d98ff4f Compare October 9, 2025 10:06

shwstppr added 11 commits October 10, 2025 10:33

Merge remote-tracking branch 'apache/main' into feature-sync-ext

d7d9a4a

more tests

4a535bd

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

missing file

0337e34

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fix line endings

c91a970

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Merge branch 'main' into feature-sync-ext

d384fb2

do not use mockConstruction

ecbdcd6

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

changes

2c70e03

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

changes for download extension

2be9d96

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fix lint and rat check

40b013d

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

add more tests

2da3004

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fix lint

fdfcfe6

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr changed the title ~~extensions: add sync functionality~~ extensions: add sync, download functionalities Oct 13, 2025

apache deleted a comment from blueorangutan Oct 13, 2025

handle when share context is disabled

942f36e

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

apache deleted a comment from blueorangutan Oct 14, 2025

github-actions bot added the status:has-conflicts label Oct 17, 2025

Merge remote-tracking branch 'apache/main' into feature-sync-ext

f923de8

github-actions bot removed the status:has-conflicts label Oct 26, 2025

changes for download via ssvm/secondary store

5525242

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

boring-cyborg bot added the component:storage label Oct 29, 2025

allow cleanup of archives on sec store

865c6fd

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

github-actions bot added the status:has-conflicts label Dec 26, 2025

Merge remote-tracking branch 'apache/main' into feature-sync-ext

de78f5b

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

github-actions bot removed the status:has-conflicts label Dec 29, 2025

github-actions bot added the status:has-conflicts label Jan 27, 2026

Merge remote-tracking branch 'apache/main' into feature-sync-ext

02c87ea

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

github-actions bot removed the status:has-conflicts label Jan 28, 2026

shwstppr added 2 commits January 30, 2026 00:33

Merge remote-tracking branch 'apache/main' into feature-sync-ext

5220a3e

fix build

35903ed

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

harikrishna-patnala reviewed Jan 30, 2026

View reviewed changes

github-actions bot added the status:has-conflicts label Jan 30, 2026

Merge remote-tracking branch 'apache/main' into feature-sync-ext

1998867

github-actions bot removed the status:has-conflicts label Jan 31, 2026

shwstppr added 2 commits January 31, 2026 13:41

secondary-storage: delete temp directory while deleting entity download

07e70f2

url Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fix

21419d1

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

extensions: add sync, download functionalities #11814

Are you sure you want to change the base?

extensions: add sync, download functionalities #11814

Uh oh!

Conversation

shwstppr commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov bot commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

github-actions bot commented Dec 26, 2025

Uh oh!

shwstppr commented Dec 30, 2025

Uh oh!

blueorangutan commented Dec 30, 2025

Uh oh!

blueorangutan commented Dec 30, 2025

Uh oh!

github-actions bot commented Jan 27, 2026

Uh oh!

borisstoyanov commented Jan 30, 2026

Uh oh!

blueorangutan commented Jan 30, 2026

Uh oh!

blueorangutan commented Jan 30, 2026

Uh oh!

harikrishna-patnala left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 30, 2026

Uh oh!

shwstppr commented Jan 31, 2026

Uh oh!

shwstppr commented Jan 31, 2026

Uh oh!

shwstppr commented Jan 31, 2026

Uh oh!

blueorangutan commented Jan 31, 2026

Uh oh!

blueorangutan commented Jan 31, 2026

Uh oh!

shwstppr commented Jan 31, 2026

Uh oh!

blueorangutan commented Jan 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

shwstppr commented Oct 9, 2025 •

edited

Loading

codecov bot commented Oct 9, 2025 •

edited

Loading