Prepare Databricks Machine Learning Professional certification exam
Databricks offers an end-to-end big data analytics and MLOps scalable solution including the ability to track, version, and manage machine learning experiments and machine learning models lifecycle.
Databricks allows the development of machine learning models using a variety of machine learning libraries, leveraging the Spark architecture for accelerated model training. It facilitates the seamless automation of training and deployment processes for new model versions in production, as well as the implementation of automated tests and ongoing monitoring of deployed models.
The Databricks Machine Learning Professional certification exam is all about assessing one’s ability to use these capabilities and perform advanced machine learning in production tasks.
Hereafter are the links to the official Databricks pages related to this exam:
- Here is the official certification information page.
- Here is the recommend official video learning path to prepare for this certification.
In this article, starting from the official Machine Learning Professional Exam Guide, are provided some tests and hands-on exercises I did while preparing the certification. Hopefully this can helps ones who wants to pass the certification to prepare the exam.
Note that some of the examples provided in the next notebooks may be inspired from the demos shown in the official video learning path.
Section 1: Experimentation
Data Management
● Read and write a Delta table
● View Delta table history and load a previous version of a Delta table
● Create, overwrite, merge, and read Feature Store tables in machine learning workflows
Advanced Experiment Tracking
● Perform MLflow experiment tracking workflows using model signatures and input examples
● Identify the requirements for tracking nested runs
● Describe the process of enabling autologging, including with the use of Hyperopt
● Log and view artifacts like SHAP plots, custom visualizations, feature data, images, and metadata
Section 2: Model Lifecycle Management
Preprocessing Logic
● Describe an MLflow flavor and the benefits of using MLflow flavors
● Describe the advantages of using the pyfunc MLflow flavor
● Describe the process and benefits of including preprocessing logic and context in custom model classes and objects
Model Management
● Describe the basic purpose and user interactions with Model Registry
● Programmatically register a new model or new model version
● Add metadata to a registered model and a registered model version
● Identify, compare, and contrast the available model stages
● Transition, archive, and delete model versions
Model Lifecycle Automation
● Identify the role of automated testing in ML CI/CD pipelines
● Describe how to automate the model lifecycle using Model Registry Webhooks and Databricks Jobs
● Identify advantages of using Job clusters over all-purpose clusters
● Describe how to create a Job that triggers when a model transitions between stages, given a scenario
● Describe how to connect a Webhook with a Job
● Identify which code block will trigger a shown webhook
● Identify a use case for HTTP webhooks and where the Webhook URL needs to come
● Describe how to list all webhooks and how to delete a webhook
Section 3: Model Deployment
Batch
● Describe batch deployment as the appropriate use case for the vast majority of deployment use cases
● Identify how batch deployment computes predictions and saves them somewhere for later use
● Identify live serving benefits of querying precomputed batch predictions
● Identify less performant data storage as a solution for other use cases
● Load registered models with load_model
● Deploy a single-node model in parallel using spark_udf
● Identify z-ordering as a solution for reducing the amount of time to read predictions from a table
● Identify partitioning on a common column to speed up querying
● Describe the practical benefits of using the score_batch operation
Streaming
● Describe Structured Streaming as a common processing tool for ETL pipelines
● Identify structured streaming as a continuous inference solution on incoming data
● Describe why complex business logic must be handled in streaming deployments
● Identify that data can arrive out-of-order with structured streaming
● Identify continuous predictions in time-based prediction store as a scenario for streaming deployments
● Convert a batch deployment pipeline inference to a streaming deployment pipeline
● Convert a batch deployment pipeline writing to a streaming deployment pipeline
Real-time
● Describe the benefits of using real-time inference for a small number of records or when fast prediction computations are needed
● Identify JIT feature values as a need for real-time deployment
● Describe model serving deploys and endpoint for every stage
● Identify how model serving uses one all-purpose cluster for a model deployment
● Query a Model Serving enabled model in the Production stage and Staging stage
● Identify how cloud-provided RESTful services in containers is the best solution for production-grade real-time deployments
Section 4: Solution and Data Monitoring
Drift Types
● Compare and contrast label drift and feature drift
● Identify scenarios in which feature drift and/or label drift are likely to occur
● Describe concept drift and its impact on model efficacy
Drift Tests and Monitoring
● Describe summary statistic monitoring as a simple solution for numeric feature drift
● Describe mode, unique values, and missing values as simple solutions for categorical feature drift
● Describe tests as more robust monitoring solutions for numeric feature drift than simple summary statistics
● Describe tests as more robust monitoring solutions for categorical feature drift than simple summary statistics
● Compare and contrast Jenson-Shannon divergence and Kolmogorov-Smirnov tests for numerical drift detection
● Identify a scenario in which a chi-square test would be useful
Comprehensive Drift Solutions
● Describe a common workflow for measuring concept drift and feature drift
● Identify when retraining and deploying an updated model is a probable solution to drift
● Test whether the updated model performs better on the more recent data
Based on my November 2023 certification experience, a solid understanding of the content presented in the official video, combined with practical hands-on tests in Databricks, proved sufficient for success. The MLflow part is definitely not to be neglected.
