Pilot 9050: fixup the preupload timeout causing resume issue #249

colorzzr · 2026-01-20T22:49:39Z

Summary

After investigation of this issue, it is caused by network timeout during the pre-upload in python version. The resumable json will treat the timeout batch AS unregistered, while backend already registered them(This issue only happened when timeout). Then mismatching happens and block resumable upload. Another PR will increase the timeout in bff-cli service. This PR is meant to fix the mismatching issue:

using newly v2 api to get items by status. If any registered item matches with manifest info, it will proceed as resumable upload.
a. old cli version will still use v1 endpoint.
fix up the manifest missing when resumable upload.

api updates:

JIRA Issues

Pilot 9050

Type of Change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update
Refactor or reformatting

Testing

Are there any new or updated tests to validate the changes?

Yes
No

Test Directions

unittest:

adding new test cases for duplication check and resumable manifest.
manual testing:
for mismatching, we can manually alter the manifest.json to create mismatching, moving registered items into unregistered fields.
for resumable manifest issue, ctrl+c the resume process. the manifest should contains correct number of informations

Versions

Pilot Release Version:
Compatible Version:

github-actions · 2026-01-20T22:51:41Z

Coverage Report

File	Stmts	Miss	Cover	Missing
__init__.py	0	0	100%
pilotcli.py	0	59	0%	5, 7–10, 12–16, 19–20, 22–23, 25–26, 28–32, 34–37, 39, 41–42, 44–45, 48, 51–57, 59, 61–64, 66–71, 74–80, 83–85
commands
__init__.py	0	0	100%
container_registry.py	0	19	59%	16, 22–24, 31–33, 41–44, 50–53, 62–65
dataset.py	0	9	89%	20, 38–39, 47, 82–83, 92, 126, 130
entry_point.py	0	11	81%	36, 42, 70, 93–95, 97–101
file.py	0	38	86%	45, 156–159, 163, 168, 171–173, 304–311, 313, 315, 329–335, 337, 366, 368, 372, 418, 421, 436–438, 444–445
folder.py	0	1	96%	18
project.py	0	2	95%	18, 61
user.py	0	9	81%	28, 50–51, 59–60, 68–69, 90–91
configs
__init__.py	0	0	100%
app_config.py	0	0	100%
config.py	0	0	100%
user_config.py	0	23	81%	33, 73, 82, 93, 126, 135–136, 139, 149, 155, 159, 163, 167, 171–172, 176, 180–181, 185, 189–190, 198, 202
utils.py	0	35	36%	13–14, 16–19, 22–23, 25, 27, 31–32, 34, 36–37, 40, 43, 45–47, 49–51, 54–57, 59, 67, 69–72, 74–75
models
__init__.py	0	0	100%
enums.py	0	0	100%
item.py	0	0	100%
service_meta_class.py	0	1	80%	9
singleton.py	0	0	100%
upload_form.py	0	4	33%	28, 40–42
resources
custom_error.py	0	0	100%
custom_help.py	0	0	100%
services
__init__.py	0	0	100%
services/clients
__init__.py	0	0	100%
base_auth_client.py	0	1	96%	53
base_client.py	0	2	95%	49, 100
services/container_registry_manager
container_registry_manager.py	0	108	16%	18–19, 22–25, 28–32, 35–43, 47–59, 61–63, 67–81, 83–85, 89–98, 100–102, 106–109, 124–135, 139–142, 146–161, 163, 165–168
services/crypto
__init__.py	0	0	100%
crypto.py	0	19	40%	35, 43–47, 49, 59–61, 69–75, 77–78
services/dataset_manager
dataset_detail.py	0	22	70%	37, 58, 65–70, 72–74, 76–85, 87
dataset_download.py	0	22	81%	51, 61–63, 72–74, 90, 92–93, 95–96, 98, 100–102, 110–113, 147, 162
dataset_list.py	0	8	79%	32–37, 39, 48
model.py	0	0	100%
services/file_manager
__init__.py	0	0	100%
file_list.py	0	12	84%	58–64, 84–86, 98, 100
file_manifests.py	0	25	78%	69, 180–194, 196, 201–205, 207–209
file_tag.py	0	40	28%	22–23, 27–31, 33, 36–48, 50–52, 54–55, 59–61, 64–68, 70–73, 75–76
services/file_manager/file_download
__init__.py	0	0	100%
download_client.py	0	125	57%	52–61, 79–80, 125–126, 128–130, 142–143, 148–149, 153–156, 158–161, 163–168, 171, 173–174, 176–178, 181, 183, 186, 190, 193–194, 196, 200, 214–216, 222–223, 232–233, 235–237, 277, 279, 295, 315–316, 321–331, 333–334, 337–338, 343–344, 348–353, 357–358, 361–363, 365–376, 383, 388, 403, 407–409, 411–424, 426
model.py	0	0	100%
services/file_manager/file_metadata
__init__.py	0	0	100%
file_metadata_client.py	0	3	95%	90–92
folder_client.py	0	4	91%	49, 78–80
services/file_manager/file_move
__init__.py	0	0	100%
file_move_client.py	0	37	57%	57–61, 63, 71, 76–80, 83–91, 93, 95–100, 111, 113–117, 119, 169–170
services/file_manager/file_trash
__init__.py	0	0	100%
file_trash_client.py	0	2	96%	65, 86
utils.py	0	3	91%	35, 59, 66
services/file_manager/file_upload
__init__.py	0	0	100%
exception.py	0	0	100%
file_upload.py	0	18	91%	94–96, 108, 136, 170–172, 419–420, 423–425, 427, 434, 455, 458, 462
models.py	0	2	96%	20, 40
upload_client.py	0	49	77%	99–102, 142, 226, 239–242, 244–257, 259, 261–265, 267–271, 273–274, 377–378, 380, 450, 456–457, 462–465, 467–468
upload_validator.py	0	13	71%	51–55, 57, 61–63, 66–67, 72, 74
services/logger_services
__init__.py	0	0	100%
debugging_log.py	0	0	100%
log_functions.py	0	0	100%
services/output_manager
__init__.py	0	0	100%
error_handler.py	0	0	100%
help_page.py	0	7	91%	13–17, 19, 21
message_handler.py	0	54	74%	24, 46, 51, 66–68, 73, 101, 106, 113, 118, 123, 127, 143, 178, 188, 197, 215, 237, 281–292, 303–308, 310–316, 327, 331–332, 336–337, 339–340, 344, 348, 352
services/project_manager
__init__.py	0	0	100%
project.py	0	6	83%	38, 44–45, 47–49
services/user_authentication
__init__.py	0	0	100%
decorator.py	0	2	93%	25, 28
token_manager.py	0	8	88%	41–45, 72–73, 90
user_login_logout.py	0	16	83%	25–26, 37, 122, 126–132, 134–136, 140–141
utils
__init__.py	0	0	100%
aggregated.py	0	21	88%	51, 61–63, 77–79, 111, 125–126, 139, 169–170, 172, 201, 209, 226–227, 271–272, 299
TOTAL	3668	840	77%

Copilot

Pull request overview

This PR fixes a resumable upload issue caused by network timeouts during pre-upload operations. When timeouts occur, the local manifest can become mismatched with the backend state, treating already-registered items as unregistered.

Changes:

Updated duplication check API from v1 to v2 endpoint to separately identify ACTIVE vs REGISTERED items
Fixed manifest key inconsistency (changed from local_path to object_path for unregistered items)
Added logic to reconcile registered items found on backend with local manifest during resume
Updated Python version support from 3.10-3.11 to 3.10-3.13
Consolidated imports and updated copyright years to 2026

Reviewed changes

Copilot reviewed 100 out of 101 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
pyproject.toml	Version bump to 3.21.0 and Python version range update
poetry.lock	Dependency updates for new Poetry version and Python support
app/services/file_manager/file_upload/upload_client.py	API v2 migration, three-way return for duplication check
app/services/file_manager/file_upload/file_upload.py	Resume logic to handle registered items mismatch
app/services/file_manager/file_upload/exception.py	Added proper exception message to super().init
tests/app/services/file_manager/file_upload/test_upload_client.py	Updated tests for new three-way return signature
tests/app/services/file_manager/file_upload/test_file_upload.py	Added comprehensive tests for manifest batch processing
Multiple files	Copyright year updates and import consolidation

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-20T22:53:00Z

app/services/file_manager/file_upload/upload_client.py

        return:
            - non_exist_file_objects(List[FileObject]): the file that need to be uploaded.
            - exist_files(List[str]): the file that has been uploaded. will be skipped
+            - [updated] registered_file_objects(List[Dict[str, Any]]): this is to handle the conner case


Spelling error: "conner case" should be "corner case".

Copilot · 2026-01-20T22:53:00Z

app/services/file_manager/file_upload/file_upload.py

-        - non_duplicate_file_objects(List[FileObject]): the list of file object that is not duplicated
+        - unregistered_items(List[FileObject]): the list of file object that is not duplicated
+        - [updated] registered_items(List[FileObject]): the list of file object that is already registered.
+            in conner case if preupload interrupted at specific batch, the local manifest


Spelling error: "conner case" should be "corner case".

Copilot · 2026-01-20T22:53:00Z

app/services/file_manager/file_upload/file_upload.py

-    unregistered_items = item_duplication_check(False, unregistered_items, upload_client)
+
+    # [updated] here duplication check api got update will filter out ACTIVE and REGISTERED items separately
+    # the reason is during the normal upload , there is a conner case that preupload got interrupted at


Spelling error: "conner case" should be "corner case".

Copilot · 2026-01-20T22:53:00Z

tests/app/services/file_manager/file_upload/test_upload_client.py

+    registered_item = FileObject('object/registered', 'local_path', 'resumable_id', 'job_id', 'item_id')

-    url = AppConfig.Connections.url_base + '/portal/v1/files/exists'
+    url = AppConfig.Connections.url_bff + '/v2/items/batch/exists'
    httpx_mock.add_response(
        method='POST',
        url=url,
-        json={'result': [dup_obj.object_path.upper() if case_insensitive else dup_obj.object_path]},
+        json={
+            'result': [
+                {'parent_path': 'object', 'name': 'duplicate', 'status': ItemStatus.ACTIVE},
+                {'parent_path': 'object', 'name': 'registered', 'status': ItemStatus.REGISTERED},
+            ]
+        },
    )

-    not_dup_list, dup_list = upload_client.check_upload_duplication([dup_obj, not_dup_object])
-    assert not_dup_list == [not_dup_object]
-    assert dup_list == [dup_obj.object_path.upper() if case_insensitive else dup_obj.object_path]
+    not_dup_list, dup_list, registered_list = upload_client.check_upload_duplication([dup_obj, not_dup_object])
+
+    assert len(not_dup_list) == 1
+    assert not_dup_list[0] == not_dup_object
+
+    assert len(dup_list) == 1
+    assert dup_list[0] == dup_obj.object_path
+
+    assert len(registered_list) == 1
+    assert registered_item.object_path == registered_list[0].get('parent_path', '') + '/' + registered_list[0].get(
+        'name', ''
+    )


The test creates a registered_item FileObject on line 316, but it is never passed to check_upload_duplication on line 330. Only dup_obj and not_dup_object are passed. The assertion on lines 339-341 that checks if registered_item.object_path matches the returned registered item appears to be checking the wrong thing - it's comparing a local variable that was never used in the function call to the API response. This test should either pass all three objects to the function or not create the unused registered_item variable.

Copilot · 2026-01-20T22:53:01Z

app/services/file_manager/file_upload/upload_client.py

        return_list = {}
        for _, item in object_path_file_object_map.items():
            return_list.update({item.object_path: item})

-        return list(object_path_file_object_map.values()), exist_files
+        return list(object_path_file_object_map.values()), active_path, registered_items


The code creates a return_list dictionary on lines 205-207, but then returns list(object_path_file_object_map.values()) on line 209 instead of using return_list. The return_list variable appears to be unused dead code. Either use list(return_list.values()) in the return statement or remove the return_list construction entirely.

colorzzr added 5 commits August 6, 2025 16:01

merged

5d99c9f

update duplicate logic to check based on item status

f846580

update COPYRIGHT

382032b

fixup the preupload timeout fails the resumable upload

9377a34

fixup the resumable upload didn't output correct batches

07f72af

colorzzr requested a review from Copilot January 20, 2026 22:49

colorzzr self-assigned this Jan 20, 2026

colorzzr added bug Something isn't working enhancement New feature or request labels Jan 20, 2026

Copilot started reviewing on behalf of colorzzr January 20, 2026 22:50 View session

Copilot AI reviewed Jan 20, 2026

View reviewed changes

colorzzr and others added 3 commits January 21, 2026 15:40

update version matrix pipeline

065e6a3

update version matrix pipeline

86e8f52

Updated docs/compatible_version.ndjson with PR #249

a9bddf3

colorzzr requested review from QXgu, gregmccoy and juyoungkimthedev January 22, 2026 21:54

gregmccoy approved these changes Jan 23, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pilot 9050: fixup the preupload timeout causing resume issue #249

Pilot 9050: fixup the preupload timeout causing resume issue #249

Uh oh!

colorzzr commented Jan 20, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Jan 20, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jan 20, 2026

Uh oh!

Copilot AI Jan 20, 2026

Uh oh!

Copilot AI Jan 20, 2026

Uh oh!

Copilot AI Jan 20, 2026

Uh oh!

Copilot AI Jan 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Pilot 9050: fixup the preupload timeout causing resume issue #249

Are you sure you want to change the base?

Pilot 9050: fixup the preupload timeout causing resume issue #249

Uh oh!

Conversation

colorzzr commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

JIRA Issues

Type of Change

Testing

Test Directions

Versions

Uh oh!

github-actions bot commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

colorzzr commented Jan 20, 2026 •

edited

Loading

github-actions bot commented Jan 20, 2026 •

edited

Loading