FP Complete


NOTE This blog post made the rounds last week before the branch was actually merged and the post was still on a review server. I’m officially publishing it as the pull request is now merged.

There is a collection of features in Stack that have been added in bit by bit, as opposed to being designed into a cohesive whole from the start. The features work, but could be a bit better. We’ve known for a while that, instead of putting in place strategic fixes, a more general refactoring of the core dependency management logic was in order. I’m happy to announce that these changes have landed in the master branch, and will be part of the next major release of Stack.

I’d like to motivate the limitations in Stack that encouraged this change, discuss the new system, mention some potential future changes, and share a few thoughts on the (very pleasant) Haskell refactoring process itself.

NOTE These features have not currently been released, so don’t try using them in a stable Stack executable. If you’d like to test them out (and I’d certainly appreciate the extra testing), you can run stack upgrade --git to build a Stack executable from the master branch.

Motivation

Consider this fairly standard snippet of a stack.yaml file:

resolver: lts-8.12
packages:
- ./site1
- ./site2
- location:
    git: https://github.com/yesodweb/yesod
    commit: 7038ae6317cb3fe4853597633ba7a40804ca9a46
  extra-dep: true
  subdirs:
  - yesod-core
  - yesod-bin
extra-deps:
- html-conduit-1.2.1.1

This is leveraging a number of features of Stack right off the bat:

This is great, but there’s a bit of pain involved in this:

As you can see, the problems aren’t insurmountable, but they are annoyances, and they seem to overlap quite a bit.

Updated syntax

Let’s rewrite that stack.yaml file to be a little bit more straightforward:

resolver: lts-8.12
packages:
- ./site1
- ./site2
extra-deps:
- html-conduit-1.2.1.1@sha256:de32ca4d6df94a7c027a11db1b2e32ef1a7ccfe0565923f24528613ade821343
- git: https://github.com/yesodweb/yesod
  commit: 7038ae6317cb3fe4853597633ba7a40804ca9a46
  subdirs:
  - yesod-core
  - yesod-bin

The first thing to notice is that the packages value is now just a list of the actual code in our project, not the dependencies.

Next, we still have html-conduit-1.2.1.1 coming from Hackage. But we have this funny @sha256:... bit at the end. This is a hash of the cabal file contents we want to use. This gives us much stronger guarantees of reproducibility than we had previously. Instead of getting whatever most recent version happens to be available, you’ll get an exact cabal file. This feature has been present for a while in Stackage snapshots, but hasn’t been accessible for local dependencies.

Next, we’ve moved the Git repo information out of packages and into extra-deps where it logically belongs. We also no longer need that extra location key. We had that so that we could also define extra-dep and subdirs keys. We now instead put the subdirs key next to the git and commit keys, and don’t need extra-dep at all (since it’s implied by being within extra-deps).

Behind the scenes, the code managing these things has changed drastically. Most importantly for our discussion here, Stack now uses the same code paths for loading up snapshots and loading up package information within the stack.yaml file. In addition to just being a good practice for keeping us sane, this means that build tool detection now works for project packages and dependencies too.

This answers a good deal of our points above (hold off for the last two when we get to custom snapshots).

Four package locations

That probably seemed like a bit of a jumble, so let’s start over. Every package has a location, which tells Stack where to get it from. Stack supports for different package locations:

All four of these have been supported in Stack since (almost) its inception. The differences now are that:

This is all well and good, but isn’t much more than a cosmetic improvement (though, in my opinion, it’s a very nice cosmetic improvement). But this gets much cooler with custom snapshots.

Custom snapshots

Stack has had some support for custom snapshots for a while, but it’s never been fully implemented, since we’ve been waiting for this extensible snapshot concept to land. Since most people aren’t very familiar with custom snapshots today, I’m not going to compare and contrast, but instead just jump in to explaining how they work now.

Stack configurations always discuss a resolver, which specifies a GHC version, a set of additional packages, build flags, and some other pieces of metadata. You’ve probably seen a few kinds of resolvers until now:

Custom snapshots answer a simple question: what if I want to define my own snapshot which isn’t LTS Haskell or Stackage Nightly? And that’s really all they are: a format for defining your own snapshots like Stackage does. However, they’ve got a number of cool features that Stackage snapshots don’t:

Let’s see how this would modify our stack.yaml from above. First, I’m going to define a my-snapshot.yaml file:

resolver: lts-8.12
name: my-snapshot # For user display only
packages:
- html-conduit-1.2.1.1@sha256:de32ca4d6df94a7c027a11db1b2e32ef1a7ccfe0565923f24528613ade821343
- git: https://github.com/yesodweb/yesod
  commit: 7038ae6317cb3fe4853597633ba7a40804ca9a46
  subdirs:
  - yesod-core
  - yesod-bin

Notice how I’ve kept the same resolver value here. What I’m stating is that I’d like my snapshot to start off with the same GHC version and package set defined in lts-8.12, and then add new packages. Next, I’ve copied my entire extra-deps section in here, and called it packages instead (since these are the packages that actually make up the snapshot, not some extra dependencies added on top).

Note that, because a custom snapshot is intended to contain immutable package data, it does not support local filepaths as package location, as these are expected to change over time.

Now the stack.yaml file:

resolver: my-snapshot.yaml
packages:
- ./site1
- ./site2

Instead of a Stackage snapshot or compiler version, my resolver now gives the path to the snapshot config file. This can be a file path, or an HTTP(S) URL. The packages section stayed the same, but my extra-deps is no longer necessary: all of my dependencies are now defined within the custom snapshot.

And in case anyone wants to get cheeky: yes, a custom snapshot can put another custom snapshot in its resolver field. You can layer these things up as many layers as you’d like. Have fun!

Just to summarize, Stack supports three different kinds of resolver values:

Where’s the global package information?

Global packages are those packages which are shipped with GHC itself (or at least end up in its global database). A funny question I bet many people never thought about is: where are global packages defined? Are they in the snapshot, or does Stack look them up from GHC itself? There are advantages both way:

Previously, global information came from the Stackage snapshots. But both because of the “possibly incorrect” reason, and because it would be a royal pain to define all of the global packages in each custom snapshot file, Stack now does something different:

Choosing between stack.yaml and custom snapshot

You may be conflicted about whether you should add extra dependencies into a stack.yaml file (as you’ve probably done until now) or define a custom snapshot. My answers may change over time with experience, but here are some good guesses:

Things we’ve lost

Besides the potential for some kind of breaking change in behavior to have crept in (NOTE please help me by testing against your projects!), the only lost features I’m aware of are:

Future changes

Here’s my biggest feature for consideration: making the project packages only support filepaths. I can think of no logical case where we’d want to support HTTP(S) or Git repos as “project packages” (meaning things that run tests, for instance). In my ideal world:

I’m not normally in favor of breaking backwards compatibility in Stack, but miscategorized extra deps has resulted in much confusion, so I’d be happy to see it go, even if it requires rewriting stack.yaml files over time.

Subscribe to our blog via email

Email subscriptions come from our Atom feed and are handled by Blogtrottr. You will only receive notifications of blog posts, and can unsubscribe any time.

Tagged