Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate if we can optimise register_asset_changes_in_db from task runner #46002

Open
1 task done
amoghrajesh opened this issue Jan 24, 2025 · 0 comments
Open
1 task done

Investigate if we can optimise register_asset_changes_in_db from task runner #46002

amoghrajesh opened this issue Jan 24, 2025 · 0 comments
Assignees
@amoghrajesh
Labels
area:datasets Issues related to the datasets feature area:task-execution-interface-aip72 AIP-72: Task Execution Interface (TEI) aka Task SDK

Comments

@amoghrajesh
Copy link
Contributor

Body

Right now, for registering asset events on task completion, we send across a lot of "events" for AssetNameRef and AssetUriRef types. Maybe we can optimise it by retrieving the Asset directly using client APIs and sending those.

This will also simplify register_asset_changes_in_db

Committer

  • I acknowledge that I am a maintainer/committer of the Apache Airflow project.
@amoghrajesh amoghrajesh added the area:task-execution-interface-aip72 AIP-72: Task Execution Interface (TEI) aka Task SDK label Jan 24, 2025
@amoghrajesh amoghrajesh self-assigned this Jan 24, 2025
@amoghrajesh amoghrajesh mentioned this issue Jan 24, 2025
Merged
@dosubot dosubot bot added the area:datasets Issues related to the datasets feature label Jan 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@amoghrajesh amoghrajesh

Labels
area:datasets Issues related to the datasets feature area:task-execution-interface-aip72 AIP-72: Task Execution Interface (TEI) aka Task SDK
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@amoghrajesh