-
Notifications
You must be signed in to change notification settings - Fork 749
[GOBBLIN-2243]Add Fallback dir for Jar caching #4160
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java
Outdated
Show resolved
Hide resolved
gobblin-yarn/src/test/java/org/apache/gobblin/yarn/GobblinYarnAppLauncherTest.java
Outdated
Show resolved
Hide resolved
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java
Outdated
Show resolved
Hide resolved
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnConfigurationKeys.java
Outdated
Show resolved
Hide resolved
| return config.withValue(GobblinYarnConfigurationKeys.JAR_CACHE_DIR, | ||
| ConfigValueFactory.fromAnyRef(fullPath)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
computing JAR_CACHE_DIR from filesystem state at config load time makes it hard to debug. We can introduce a JarCachePathResolver instead that computes the resolved cache dir, when needed.
If that's too big a change, for now, let's check that JAR_CACHE_DIR is not already explicitly configured(if it is, use it as is) to avoid overriding an existing config.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
makes sense, removed logic from constructor , made JarCachePathResolver
a3a1a43 to
098498a
Compare
098498a to
e83b2b4
Compare
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA
Description
Add Jar Cache Directory Validation with Fallback
Summary
Implements validation for jar cache directories with automatic fallback handling in GobblinYarnAppLauncher.
Changes
Added addJarCachingConfig() method: Validates JAR_CACHE_DIR exists on filesystem, falls back to FALLBACK_JAR_CACHE_DIR if not found, or disables jar caching if neither exists
Tests
Added unit tests
Commits