Get Started with DVC (2024)

By clicking play, you agree to YouTube's Privacy Policy and Terms of Service

How cool would it be to track large datasets and machine learning modelsalongside your code, sidestepping all the limitations of storing it in Git?Imagine cloning a repository and immediately seeing your datasets, checkpointsand models staged in your workspace. Imagine switching to a different version ofa 100Gb file in less than a second with a git checkout.

💫 DVC is your "Git for data"!

Initializing a project

Before we begin, settle on a directory for this guide. Everything we will dowill be self contained there.

Imagine we want to build an ML project from scratch. Let's start by creating aGit repository:

$ mkdir example-get-started$ cd example-get-started$ git init

This directory name is used in ourexample-get-started repo.

Inside your chosen directory, we will use our current working directory as aDVC project. Let's initialize it by running dvc init inside a Gitproject:

$ dvc init

A few internal files arecreated that should be added to Git:

$ git statusChanges to be committed: new file: .dvc/.gitignore new file: .dvc/config ...$ git commit -m "Initialize DVC"

Now you're ready to DVC!

Tracking data

Working inside an initialized projectdirectory, let's pick a piece of data to work with. We'll use an exampledata.xml file, though any text or binary file (or directory) will do. Start byrunning:

$ dvc get https://github.com/iterative/dataset-registry \ get-started/data.xml -o data/data.xml

We used dvc get above to show how DVC can turn any Git repo into a "dataregistry". dvc get can download any file or directory tracked in a DVC repository.

Use dvc add to start tracking the dataset file:

$ dvc add data/data.xml

DVC stores information about the added file in a special .dvc file nameddata/data.xml.dvc. This small, human-readable metadata file acts as aplaceholder for the original data for the purpose of Git tracking.

Next, run the following commands to track changes in Git:

$ git add data/data.xml.dvc data/.gitignore$ git commit -m "Add raw data"

Now the metadata about your data is versioned alongside your source code,while the original data file was added to .gitignore.

dvc add moved the data to the project's cache, andlinked it back to the workspace. The .dvc/cache willlook like this:

.dvc/cache/files/md5└── 22 └── a1a2931c8370d3aeedd7183606fd7f

The hash value of the data.xml file we just added (22a1a29...) determinesthe cache path shown above. And if you check data/data.xml.dvc, you will findit there too:

outs: - md5: 22a1a2931c8370d3aeedd7183606fd7f path: data.xml

Storing and sharing

You can upload DVC-tracked data to a variety of storage systems (remote orlocal) referred to as"remotes". For simplicity, wewill use a "local remote" for this guide, which is just a directory in the localfile system.

Configuring a remote

Before pushing data to a remote we need to set it up using the dvc remote addcommand:

$ mkdir /tmp/dvcstore$ dvc remote add -d myremote /tmp/dvcstore

$ mkdir %TEMP%/dvcstore$ dvc remote add -d myremote %TEMP%\dvcstore

DVC supports many remote storage types, including Amazon S3, NFS, SSH, Google Drive,Azure Blob Storage, and HDFS.

An example for a common use case is configuring an Amazon S3 remote:

$ dvc remote add -d storage s3://mybucket/dvcstore

For this to work, you'll need an AWS account and credentials set up to allowaccess.

To learn more about storage remotes, see the Remote Storage Guide.

Uploading data

Now that a storage remote was configured, run dvc push to upload data:

$ dvc push

dvc push copied the data cached locally to the remote storage weset up earlier. The remote storage directory should look like this:

.../dvcstore/files/md5└── 22 └── a1a2931c8370d3aeedd7183606fd7f

Usually, we would also want to Git track any code changes that led to the datachange ( git add, git commit and git push ).

Retrieving data

Once DVC-tracked data and models are stored remotely, they can be downloadedwith dvc pull when needed (e.g. in other copies of this project).Usually, we run it after git pull or git clone.

Let's try this now:

$ dvc pull

After running dvc push above, the dvc pull command afterwards wasshort-circuited by DVC for efficiency. The project's data/data.xml file, ourcache and the remote storage were all already in sync. We need toempty the cache and delete data/data.xml from our project if wewant to have DVC actually moving data around. Let's do that now:

$ rm -rf .dvc/cache$ rm -f data/data.xml

$ rmdir .dvc\cache$ del data\data.xml

Now we can run dvc pull to retrieve the data from the remote:

$ dvc pull

Making local changes

Next, let's say we obtained more data from some external source. We willsimulate this by doubling the dataset contents:

$ cp data/data.xml /tmp/data.xml$ cat /tmp/data.xml >> data/data.xml

$ copy data\data.xml %TEMP%\data.xml$ type %TEMP%\data.xml >> data\data.xml

After modifying the data, run dvc add again to track the latest version:

$ dvc add data/data.xml

Now we can run dvc push to upload the changes to the remote storage, followedby a git commit to track them:

$ dvc push$ git commit data/data.xml.dvc -m "Dataset updates"

Switching between versions

A commonly used workflow is to use git checkout to switch to a branch orcheckout a specific .dvc file revision, followed by a dvc checkout to syncdata into your workspace:

$ git checkout <...>$ dvc checkout

Return to a previous version of the dataset

Let's go back to the original version of the data:

$ git checkout HEAD~1 data/data.xml.dvc$ dvc checkout

Let's commit it (no need to do dvc push this time since this original versionof the dataset was already saved):

$ git commit data/data.xml.dvc -m "Revert dataset updates"

As you can see, DVC is technically not a version control system by itself! Itmanipulates .dvc files, whose contents define the data file versions. Git isalready used to version your code, and now it can also version your dataalongside it.

Following This Guide

To help you understand and use DVC better, consider the following threeuse-cases: data pipelines, experiment tracking and model management.You may pick any to start learning about how DVC helps you "solve" thatscenario!

Choose a trail to jump into its first chapter:

  • Data Pipelines - Use DVC as a build system for reproducible, data drivenpipelines.

  • Experiment Management - Easily track your experiments and their progressby only instrumenting your code, and collaborate on ML experiments like softwareengineers do for code.

  • Model Registry - Use the DVC model registry to manage the lifecycle of yourmodels in an auditable way. Easily access your models and integrate your modelregistry actions into CICD pipelines to follow GitOps best practices.

Get Started with DVC (2024)
Top Articles
Latest Posts
Recommended Articles
Article information

Author: Annamae Dooley

Last Updated:

Views: 6206

Rating: 4.4 / 5 (65 voted)

Reviews: 80% of readers found this page helpful

Author information

Name: Annamae Dooley

Birthday: 2001-07-26

Address: 9687 Tambra Meadow, Bradleyhaven, TN 53219

Phone: +9316045904039

Job: Future Coordinator

Hobby: Archery, Couponing, Poi, Kite flying, Knitting, Rappelling, Baseball

Introduction: My name is Annamae Dooley, I am a witty, quaint, lovely, clever, rich, sparkling, powerful person who loves writing and wants to share my knowledge and understanding with you.