To Git or Not to Git – Binary Files and Git Attributes (Part I)

We’ve talked about it in the second part of the introductory module of the Version Control Basics course. There are use cases where version control and Git don’t deliver their full value or might not be the right tools at all. Binary files are one such use case, especially larger binary files, which are usually better left to systems like Git LFS anyway.

Since Git can’t easily track changes line by line in binary files, it doesn’t get to show what it can do when it comes to storing efficiently or showing changes meaningfully. Changing and committing binary files frequently will only bloat your repository and slow down your operations; diffs won’t help you much either, at least not out of the box. That said, not all binary files are “equally” binary.

The binary spectrum

There are several flavors of “binary-ness”:

Mostly binary files (like media files) – usually don’t belong in Git unless small and rarely-changing, like a logo.
Hybrid files (text with binary chunks, like PDF, or in a binary wrapper, like .docx files) – may belong in Git, but with catches, e.g., if small and you commit less frequently.
Essentially text files Git thinks are binary, often because it finds a NUL byte in the file sample it looks at – these belong in Git, unless they are text files where versioning doesn’t make sense, like machine-generated artifacts.

Would you rather watch? This part and the basics of Binary Files and Git LFS are covered in this video:

Treating files as text

If you need Git to see your files as text, there are ways to do so. For instance, you can define Git attributes for each file (type) and purpose, i.e., attribute, you want Git to treat your file as text:

*.myFile text     # enable end-of-line conversion
*.myFile diff     # enable textual diffs
*.myFile merge    # enable three-way merging

Most of the time, you want to define these Git attributes in a .gitattributes file (usually in the same repo root where your .gitignore file lives), to be able to version these settings. But you can also use a .git/info/attributes file when these rules are just your personal preferences you don’t want to impose on others.

You can also set any of these attributes independently of each other. For instance, *.myFile text sets just the attribute text to normalize line endings for all files of type myFile by

converting them to Unix line endings (LF) in the index (so when using git add)
possibly converting them to Windows CRLF or Unix LF endings in your working directory, to match your setup.

Setting this attribute is useful when working with collaborators on different platforms (Windows vs. Mac/Linux), because normalizing line endings will not make files look like they have changed just because of line ending differences.

But when you set just the text attribute, Git will use its own heuristics for other attributes like diff (to decide whether it should output a line-by-line comparison like for text files instead of just telling you that the binary files differ) or merge (to attempt a line-based three-way merge or report there is a conflict).

Explicitly treating files as binary

Conversely, you can unset these attributes for files you don’t want Git to enable rules for end-of-line conversion, textual diffs, or the usual three-way merge for text files.

A rule like *.myFile -diff can be useful for files that are technically text, but where diffs may end up hard to read – this may happen, for instance, when you edit files by pointing-and-clicking in a GUI tool rather than manually.

A more compact alternative for unsetting all three attributes at once is by setting the macro attribute binary:

*.myFile binary

Telling Git how to diff properly

Git attributes can do a few more things. Another relatively common use case is defining custom diff drivers to extract the text to use for diffing purposes and help you better understand what changed. Note this will only change the output of git diff, not the files themselves.

To use a custom diff driver, you set the diff attribute to a string:

*.myFile diff=customDiff

The string you set the diff attribute to is arbitrary, but you need to use it consistently afterwards when configuring Git to define which conversion program is necessary for the custom comparison. You also have to make sure the conversion program is installed.

To configure Git to use the conversion program that extracts readable text from the file for diffing purposes, you can, for instance, run git config twice:

Once with the textconv option to tell Git how to extract the readable text from the file:

git config diff.customDiff.textconv "converter command"

A second time to have the file still treated as binary when not diffing:

git config diff.customDiff.binary true

Depending on your setup, you might get readable diffs for some binary files out of the box. As an example, Git for Windows uses astextplain to extract text from .docx or PDF files. You then only need to configure Git if you want to use other tools or other setups. For instance, to custom diff .docx files with Pandoc, you could proceed as follows:

set the diff attribute in your .gitattributes file to *.docx diff=docx
configure Git to define the custom diff driver, e.g.,

git config diff.docx.textconv "pandoc --to=markdown"
git config diff.docx.binary true

Custom diffing text files

Custom comparison tools can also be useful for text files, for instance, to help customize hunk headers. A hunk header is a label telling you where in the file a hunk (a contiguous block of changes in your file) occurs.

When it comes to text files, Git already has built-in patterns for common languages like Python, TeX, Markdown, etc. In such cases, you can simply set the diff attribute to the built-in pattern of interest to have more helpful hunk headers.

For instance, explicitly setting *.py diff=python tells Git to use the Python pattern when diffing .py files. Depending on your setup, your hunk headers should display line range info and potentially the enclosing function or class (if any). Should this not work out of the box, explicitly setting the custom diff driver ensures Git will show the enclosing function or class definition, if there is one.

If you want more customization or work with file types without built-in patterns, you’ll need to configure Git accordingly.

In a nutshell

Some binary files are “more binary” than others and you can use Git attributes (and maybe git config) to make better use of Git’s capabilities.