Skip to content

Conversation

@jveitchmichaelis
Copy link
Collaborator

@jveitchmichaelis jveitchmichaelis commented Dec 13, 2025

Checked if upstream fixes allow training DETR or if the CI is still breaking. Cause seems to be garbage weight initialisation by transformers, which caused extreme gradients that failed from-scratch training (nan boxes in output). This would then fail the low level tests in the CI, but shouldn't affect using pre-trained weights.

This seems to be fixed and released now, and the CI is pulling 5.0.0. This only affected a narrow use-case and should not matter to most users, so if someone hasn't updated, it shouldn't break other things.

I don't think it's necessary to restrict transformers versions in pyproject as everything else should work fine with 4.57, but we could include that (e.g. transformers<4.57, transformers>=5.0.0)

@codecov
Copy link

codecov bot commented Jan 30, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.16%. Comparing base (3146b96) to head (8be81c2).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1231      +/-   ##
==========================================
- Coverage   87.89%   87.16%   -0.73%     
==========================================
  Files          20       20              
  Lines        2776     2782       +6     
==========================================
- Hits         2440     2425      -15     
- Misses        336      357      +21     
Flag Coverage Δ
unittests 87.16% <ø> (-0.73%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jveitchmichaelis jveitchmichaelis changed the title wip: test unblock transformers Unpin transformers for v5 Jan 30, 2026
@jveitchmichaelis jveitchmichaelis marked this pull request as ready for review January 30, 2026 18:42
@jveitchmichaelis jveitchmichaelis changed the title Unpin transformers for v5 Unpin transformers for v5.0.0 Jan 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant