Skip to content

Pilot 9050: fixup the preupload timeout causing resume issue #249

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
colorzzr wants to merge 8 commits into
base: develop
Choose a base branch
from
Open

Pilot 9050: fixup the preupload timeout causing resume issue #249

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
5d99c9f
merged
colorzzr Aug 6, 2025
f846580
update duplicate logic to check based on item status
colorzzr Jan 20, 2026
382032b
update COPYRIGHT
colorzzr Jan 20, 2026
9377a34
fixup the preupload timeout fails the resumable upload
colorzzr Jan 20, 2026
07f72af
fixup the resumable upload didn't output correct batches
colorzzr Jan 20, 2026
065e6a3
update version matrix pipeline
colorzzr Jan 21, 2026
86e8f52
update version matrix pipeline
colorzzr Jan 21, 2026
a9bddf3
Updated docs/compatible_version.ndjson with PR #249
Jan 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 19 additions & 14 deletions .github/workflows/update_compatible_version.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,33 +39,38 @@ jobs:
- name: Extract Versions from PR
if: ${{ steps.check_version.outputs.current != steps.check_version.outputs.last }}
id: extract_versions
shell: bash
run: |

PR_BODY="${{ github.event.pull_request.body }}"
# Dump PR body to a file WITHOUT bash interpreting backticks / $(...)
cat > pr_body.md <<'__PR_BODY__'
${{ github.event.pull_request.body }}
__PR_BODY__

# Extract the "## Versions" section
echo "$PR_BODY" | awk '/## Versions/{flag=1; next} /^## /{flag=0} flag' > PR_VERSIONS.md
awk '/## Versions/{flag=1; next} /^## /{flag=0} flag' pr_body.md > PR_VERSIONS.md

CLI_VERSION="$(grep -oP '(?<=- Pilot Release Version: ).*' PR_VERSIONS.md 2>/dev/null || true)"
CLI_VERSION="${CLI_VERSION//$'\r'/}"

# Read extracted content
CLI_VERSION=$(grep -oP '(?<=- Pilot Release Version: ).*' PR_VERSIONS.md | tr -d '\r')
COMPATIBLE_VERSION=$(grep -oP '(?<=- Compatible Version: ).*' PR_VERSIONS.md | tr -d '\r')
COMPATIBLE_VERSION="$(grep -oP '(?<=- Compatible Version: ).*' PR_VERSIONS.md 2>/dev/null || true)"
COMPATIBLE_VERSION="${COMPATIBLE_VERSION//$'\r'/}"

# Handle missing values
CLI_VERSION=${CLI_VERSION:-"Unknown"}
COMPATIBLE_VERSION=${COMPATIBLE_VERSION:-"Unknown"}

# Get the latest Git tag (assuming it's the CLI version)
LATEST_CLI_VERSION=$(grep '^version =' pyproject.toml | sed -E 's/version = "(.*)"/\1/')
LAST_VERSION="$(
tail -n 1 docs/compatible_version.ndjson 2>/dev/null \
| jq -r '."Cli Version"' 2>/dev/null \
|| true
)"
LAST_VERSION=${LAST_VERSION:-""}

# Convert to NDJSON format
NEW_ENTRY="{\"Cli Version\": \"$LATEST_CLI_VERSION\", \"Pilot Release Version\": \"$CLI_VERSION\", \"Compatible Version\": \"$COMPATIBLE_VERSION\"}"
NEW_ENTRY="{\"Cli Version\": \"$LAST_VERSION\", \"Pilot Release Version\": \"$CLI_VERSION\", \"Compatible Version\": \"$COMPATIBLE_VERSION\"}"

# Ensure file exists before appending
touch docs/compatible_version.ndjson

# Append new entry to NDJSON file
echo "$NEW_ENTRY" >> docs/compatible_version.ndjson


- name: Embed Compatible Versions into README
if: ${{ steps.check_version.outputs.current != steps.check_version.outputs.last }}
run: |
Expand Down
43 changes: 12 additions & 31 deletions .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,19 +1,16 @@
repos:

- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.3.0
rev: v6.0.0
hooks:
- id: check-added-large-files
- id: check-docstring-first
- id: check-merge-conflict
- id: check-toml
- id: check-yaml
- id: double-quote-string-fixer
- id: end-of-file-fixer
- id: trailing-whitespace

- repo: https://github.com/psf/black
rev: 22.8.0
rev: 26.1.0
hooks:
- id: black
args: [
Expand All @@ -22,51 +19,35 @@ repos:
]

- repo: https://github.com/PyCQA/isort
rev: 5.12.0
rev: 7.0.0
hooks:
- id: isort
args: [
'--line-length=120',
'--profile=black',
'--filter-files',
'--force-single-line-imports',
'--reverse-relative',
]

- repo: https://github.com/PyCQA/flake8
rev: 5.0.4
rev: 7.3.0
hooks:
- id: flake8
additional_dependencies: [
'pycodestyle==2.9.1', # E,W
'pyflakes==2.5.0', # F
'mccabe==0.7.0', # C
'flake8-bugbear==22.9.11', # B
'flake8-builtins==1.5.3', # A
'flake8-comprehensions==3.10.0', # C4
'flake8-debugger==4.1.2', # T1
'flake8-logging-format==0.7.5', # G
'flake8-print==5.0.0', # T2
'flake8-bugbear', # B
'flake8-builtins', # A
'flake8-comprehensions', # C4
'flake8-debugger', # T1
'flake8-print', # T2
]
args: [
'--select=E,W,F,C,B,A,C4,T1,G,T2',
'--ignore=E203,W503,B008,B305,A003,G004',
'--select=E,W,F,C,B,A,C4,T1,T2',
'--ignore=E203,W503,B008,B305,A003',
'--max-complexity=10',
'--max-line-length=120',
]

# - repo: https://github.com/myint/docformatter
# rev: v1.5.0
# hooks:
# - id: docformatter
# args: [
# '--wrap-summaries=120',
# '--wrap-descriptions=120',
# '--in-place',
# ]

- repo: https://github.com/Lucas-C/pre-commit-hooks
rev: v1.4.2
rev: v1.5.6
hooks:
- id: insert-license
files: \.py$
Expand Down
3 changes: 2 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,8 @@ Command line tool that allows the user to execute data operations on the platfor
| 3.19.6 | 2.15.2 | 2.15.2 |
| 3.19.7 | 2.15.2 | 2.15.2 |
| 3.20.0 | 2.16.0 | 2.15.2 |
| 3.20.1 | 2.16 | 2.16 |
| 3.20.1 | 2.16.0 | 2.16.0 |
| 3.20.1 | Unknown | Unknown |
<!-- COMPATIBLE_VERSIONS_END -->

## Build Instructions
Expand Down
2 changes: 1 addition & 1 deletion app/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
# Copyright (C) 2022-2025 Indoc Systems
# Copyright (C) 2022-2026 Indoc Systems
#
# Contact Indoc Systems for any questions regarding the use of this source code.
2 changes: 1 addition & 1 deletion app/commands/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
# Copyright (C) 2022-2025 Indoc Systems
# Copyright (C) 2022-2026 Indoc Systems
#
# Contact Indoc Systems for any questions regarding the use of this source code.
2 changes: 1 addition & 1 deletion app/commands/container_registry.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (C) 2022-2025 Indoc Systems
# Copyright (C) 2022-2026 Indoc Systems
#
# Contact Indoc Systems for any questions regarding the use of this source code.

Expand Down
2 changes: 1 addition & 1 deletion app/commands/dataset.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (C) 2022-2025 Indoc Systems
# Copyright (C) 2022-2026 Indoc Systems
#
# Contact Indoc Systems for any questions regarding the use of this source code.

Expand Down
35 changes: 15 additions & 20 deletions app/commands/entry_point.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (C) 2022-2025 Indoc Systems
# Copyright (C) 2022-2026 Indoc Systems
#
# Contact Indoc Systems for any questions regarding the use of this source code.

Expand All @@ -8,29 +8,24 @@

from app.services.user_authentication.decorator import require_login_session

from .container_registry import create_project
from .container_registry import get_secret
from .container_registry import invite_member
from .container_registry import list_projects
from .container_registry import list_repositories
from .dataset import dataset_download
from .dataset import dataset_list
from .dataset import dataset_show_detail
from .file import file_check_manifest
from .file import file_download
from .file import file_export_manifest
from .file import file_list
from .file import file_metadata_download
from .file import file_move
from .file import file_put
from .file import file_resume
from .file import file_trash
from .container_registry import create_project, get_secret, invite_member, list_projects, list_repositories
from .dataset import dataset_download, dataset_list, dataset_show_detail
from .file import (
file_check_manifest,
file_download,
file_export_manifest,
file_list,
file_metadata_download,
file_move,
file_put,
file_resume,
file_trash,
)
from .folder import folder_create

# Import custom commands
from .project import project_list_all
from .user import login
from .user import logout
from .user import login, logout

container_registry_enabled = os.environ.get('PILOT_CLI_CONTAINER_REGISTRY_ENABLED', 'false') == 'true'

Expand Down
43 changes: 20 additions & 23 deletions app/commands/file.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,45 +1,42 @@
# Copyright (C) 2022-2025 Indoc Systems
# Copyright (C) 2022-2026 Indoc Systems
#
# Contact Indoc Systems for any questions regarding the use of this source code.

import ast
import json
import os
from sys import exit
import sys

import click
from click.exceptions import Abort

import app.services.output_manager.help_page as file_help
import app.services.output_manager.message_handler as message_handler
from app.configs.app_config import AppConfig
from app.models.item import ItemStatus
from app.models.item import ItemType
from app.models.item import ItemStatus, ItemType
from app.services.file_manager.file_download.download_client import SrvFileDownload
from app.services.file_manager.file_list import SrvFileList
from app.services.file_manager.file_manifests import SrvFileManifests
from app.services.file_manager.file_metadata.file_metadata_client import FileMetaClient
from app.services.file_manager.file_move.file_move_client import FileMoveClient
from app.services.file_manager.file_trash.file_trash_client import FileTrashClient
from app.services.file_manager.file_trash.utils import parse_trash_paths
from app.services.file_manager.file_upload.file_upload import assemble_path
from app.services.file_manager.file_upload.file_upload import resume_upload
from app.services.file_manager.file_upload.file_upload import simple_upload
from app.services.file_manager.file_upload.file_upload import assemble_path, resume_upload, simple_upload
from app.services.file_manager.file_upload.upload_validator import UploadEventValidator
from app.services.logger_services.debugging_log import debug_logger
from app.services.output_manager.error_handler import ECustomizedError
from app.services.output_manager.error_handler import SrvErrorHandler
from app.services.output_manager.error_handler import customized_error_msg
from app.services.output_manager.error_handler import ECustomizedError, SrvErrorHandler, customized_error_msg
from app.services.user_authentication.decorator import require_valid_token
from app.utils.aggregated import doc
from app.utils.aggregated import fit_terminal_width
from app.utils.aggregated import get_file_info_by_geid
from app.utils.aggregated import get_zone
from app.utils.aggregated import identify_target_folder
from app.utils.aggregated import normalize_input_paths
from app.utils.aggregated import normalize_join
from app.utils.aggregated import remove_the_output_file
from app.utils.aggregated import search_item
from app.utils.aggregated import (
doc,
fit_terminal_width,
get_file_info_by_geid,
get_zone,
identify_target_folder,
normalize_input_paths,
normalize_join,
remove_the_output_file,
search_item,
)


@click.command()
Expand Down Expand Up @@ -159,7 +156,7 @@ def file_put(**kwargs): # noqa: C901
pass
except Abort:
message_handler.SrvOutPutHandler.cancel_upload()
exit(1)
sys.exit(1)

# check if user input at least one file/folder
if len(files) == 0:
Expand All @@ -173,7 +170,7 @@ def file_put(**kwargs): # noqa: C901
)
except Abort:
message_handler.SrvOutPutHandler.cancel_upload()
exit(1)
sys.exit(1)

project_code, folder_type, target_folder = identify_target_folder(project_path)
upload_val_event = {
Expand Down Expand Up @@ -533,12 +530,12 @@ def file_move(**kwargs):
message_handler.SrvOutPutHandler.move_action_failed(
src_item_path, dest_item_path, 'Cannot move files between different projects'
)
exit(1)
sys.exit(1)
elif len(src_item.split('/')) <= 2 or len(dest_item.split('/')) <= 1:
message_handler.SrvOutPutHandler.move_action_failed(
src_item_path, dest_item_path, 'Cannot move root/name/shared folders'
)
exit(1)
sys.exit(1)

# tranlate keyword to correct object path
src_keyword, src_path = src_item.split('/', 1)
Expand Down
13 changes: 4 additions & 9 deletions app/commands/folder.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (C) 2022-2025 Indoc Systems
# Copyright (C) 2022-2026 Indoc Systems
#
# Contact Indoc Systems for any questions regarding the use of this source code.

Expand All @@ -7,14 +7,9 @@
import app.services.output_manager.message_handler as message_handler
from app.configs.app_config import AppConfig
from app.services.file_manager.file_metadata.folder_client import FolderClient
from app.services.output_manager.error_handler import ECustomizedError
from app.services.output_manager.error_handler import SrvErrorHandler
from app.services.output_manager.help_page import FileHELP
from app.services.output_manager.help_page import FolderHelp
from app.services.output_manager.help_page import file_help_page
from app.services.output_manager.help_page import folder_help_page
from app.utils.aggregated import doc
from app.utils.aggregated import validate_folder_name
from app.services.output_manager.error_handler import ECustomizedError, SrvErrorHandler
from app.services.output_manager.help_page import FileHELP, FolderHelp, file_help_page, folder_help_page
from app.utils.aggregated import doc, validate_folder_name


@click.command()
Expand Down
2 changes: 1 addition & 1 deletion app/commands/project.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (C) 2022-2025 Indoc Systems
# Copyright (C) 2022-2026 Indoc Systems
#
# Contact Indoc Systems for any questions regarding the use of this source code.

Expand Down
16 changes: 8 additions & 8 deletions app/commands/user.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (C) 2022-2025 Indoc Systems
# Copyright (C) 2022-2026 Indoc Systems
#
# Contact Indoc Systems for any questions regarding the use of this source code.

Expand All @@ -13,13 +13,13 @@
import app.services.output_manager.message_handler as mhandler
from app.models.enums import LoginMethod
from app.services.user_authentication.decorator import require_login_session
from app.services.user_authentication.user_login_logout import login_using_api_key
from app.services.user_authentication.user_login_logout import user_device_id_login
from app.services.user_authentication.user_login_logout import user_logout
from app.services.user_authentication.user_login_logout import validate_user_device_login
from app.utils.aggregated import doc
from app.utils.aggregated import get_latest_cli_version
from app.utils.aggregated import get_version_compatibility
from app.services.user_authentication.user_login_logout import (
login_using_api_key,
user_device_id_login,
user_logout,
validate_user_device_login,
)
from app.utils.aggregated import doc, get_latest_cli_version, get_version_compatibility


@click.command()
Expand Down
2 changes: 1 addition & 1 deletion app/configs/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
# Copyright (C) 2022-2025 Indoc Systems
# Copyright (C) 2022-2026 Indoc Systems
#
# Contact Indoc Systems for any questions regarding the use of this source code.
2 changes: 1 addition & 1 deletion app/configs/app_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (C) 2022-2025 Indoc Systems
# Copyright (C) 2022-2026 Indoc Systems
#
# Contact Indoc Systems for any questions regarding the use of this source code.

Expand Down
5 changes: 2 additions & 3 deletions app/configs/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (C) 2022-2025 Indoc Systems
# Copyright (C) 2022-2026 Indoc Systems
#
# Contact Indoc Systems for any questions regarding the use of this source code.

Expand All @@ -8,8 +8,7 @@
from typing import Set

from pydantic import computed_field
from pydantic_settings import BaseSettings
from pydantic_settings import SettingsConfigDict
from pydantic_settings import BaseSettings, SettingsConfigDict


class Settings(BaseSettings):
Expand Down
Loading