A data engineering team is building a production Databricks Job that uses a custom internal Python library (mycompany-utils==2.3.1) published to a private PyPI repository, plus two public PyPI packages: pandas==2.0.3 and scikit-learn==1.4.0. The job runs on a Job Cluster with Databricks Runtime 14.3 LTS. The team has three requirements: (1) the custom library must be version-pinned to prevent accidental upgrades, (2) all dependencies must be available at the start of the first task, (3) the solution must work in a CI/CD pipeline without manual steps. Which cluster library configuration approach correctly satisfies all three requirements?
Answer & explanation
Correct answer: B
WHY B: Configuring libraries in the DAB job YAML libraries block is the correct, CI/CD-native approach. Libraries declared at the cluster level in a job definition are installed during cluster initialization — before any task runs — ensuring availability from task one. Version pinning is expressed directly in the YAML (e.g., pypi: {package: 'pandas==2.0.3'}), and the YAML is version-controlled in Git. Private PyPI repos are supported via the repo field or workspace-configured secret-backed repo credentials. WHY NOT A: %pip install in a notebook restarts the Python kernel after installation, which can cause issues with shared cluster state and does not guarantee availability before other tasks. For multi-task jobs, each task notebook would need its own install command, introducing duplication and fragility. WHY NOT C: Using an all-purpose cluster violates the job isolation principle, and managing library versions through the UI is not CI/CD-compatible. WHY NOT D: Custom Docker containers require Databricks Runtime for Machine Learning or a custom container policy, which is not guaranteed for standard LTS runtimes. This approach is heavier than necessary and adds container build/publish complexity. WHY NOT E: spark.databricks.pip.requirements referencing a DBFS path is not a standard or officially supported cluster initialization mechanism for pip dependencies. The correct supported methods are cluster library configuration and %pip install.