There's a common pattern in Rust APIs: returning a relatively complex data type which provide a trait implementation we want to work with. One of the first places many Rust newcomers encounter this is with iterators. For example, if I want to provide a function that returns the range of numbers 1 to 10, it may look like this:
use std::ops::RangeInclusive;
fn one_to_ten() -> RangeInclusive<i32> {
1..=10i32
}
This obscures the iterator-ness of what's happening here. However, the situation gets worse as you start making things more complicated, e.g.:
use std::iter::Filter;
fn is_even(x: &i32) -> bool {
x % 2 == 0
}
fn evens() -> Filter<RangeInclusive<i32>, for<'r> fn(&'r i32) -> bool> {
one_to_ten().filter(is_even)
}
Or even more crazy:
use std::iter::Map;
fn double(x: i32) -> i32 {
x * 2
}
fn doubled() ->
Map<
Filter<
RangeInclusive<i32>,
for<'r> fn(&'r i32) -> bool
>,
fn(i32) -> i32
> {
evens().map(double)
}
This is clearly not the code we want to write! Fortunately, we now have a more elegant way to state our intention: impl Trait. This feature allows us to say that a function returns a value which is an implementation of some trait, without needing to explicitly state the concrete type. We can rewrite the signatures above with:
fn one_to_ten() -> impl Iterator<Item = i32> {
1..=10i32
}
fn is_even(x: &i32) -> bool {
x % 2 == 0
}
fn evens() -> impl Iterator<Item = i32> {
one_to_ten().filter(is_even)
}
fn double(x: i32) -> i32 {
x * 2
}
fn doubled() -> impl Iterator<Item = i32> {
evens().map(double)
}
fn main() {
for x in doubled() {
println!("{}", x);
}
}
This can be a boon for development, especially when we get to more complicated cases (like futures and tokio heavy code). However, I'd like to present one case where impl Trait demonstrates a limitation. Hopefully this will help explain some of the nuances of ownership and its interaction with this feature.
Introducing the riddle
Have a look at this code, which does not compile:
// Try replacing with (_: &String)
fn make_debug<T>(_: T) -> impl std::fmt::Debug {
42u8
}
fn test() -> impl std::fmt::Debug {
let value = "value".to_string();
// try removing the ampersand to get this to compile
make_debug(&value)
}
pub fn main() {
println!("{:?}", test());
}
In this code, we have a make_debug function, which takes any value at all, entirely ignores that value, and returns a u8. However, instead of including the u8 in the function signature, I say impl Debug (which is fully valid: u8 does in fact implement Debug). The test function produces its own impl Debug by passing in a &String to make_debug.
When I try to compile this, I get the error message:
error[E0597]: `value` does not live long enough
--> src/main.rs:10:16
|
6 | fn test() -> impl std::fmt::Debug {
| -------------------- opaque type requires that `value` is borrowed for `'static`
...
10 | make_debug(&value)
| ^^^^^^ borrowed value does not live long enough
11 | }
| - `value` dropped here while still borrowed
Before we try to understand this error message, I want to deepen the riddle here. There are a large number of changes I can make to this code to get it to compile. For example:
- If I replace the
T parameter on make_debug with &String (or the more idiomatic &str). the code compiles. For some reason, being polymorphic causes a problem.
- Perhaps even stranger, changing the signature from
make_debug<T>(_: T) to make_debug<T>(_: &T) fixes it too! What's weird about this is that T allows references to be passed in, so why does &T fix anything?
- And finally, in the call to
make_debug, if we pass the value (via a move) instead of a reference to the value, everything compiles, e.g. make_debug(value) instead of make_debug(&value). At least intuitively, I would expect to get less lifetime errors when using references.
Something subtle is going on here, let's try to understand it, bit by bit.
Lifetimes with concrete types
Let's simplify our make_debug function to explicitly take a String:
fn make_debug(_: String) -> impl std::fmt::Debug {
42u8
}
What's the lifetime of that parameter? Well, make_debug consumes the value completely and then drops it. The value cannot be used outside of the function any more. Interestingly though, the fact that make_debug drops it is not really reflected in the type signature of the function; it just says we return an impl Debug. To prove the point a bit, we can instead return the parameter itself instead of our 42u8:
fn make_debug(message: String) -> impl std::fmt::Debug {
//42u8
message
}
In this case, the ownership of the message transfers from the make_debug function itself to the returned impl Debug value. That's an interesting and important observation which we'll get back to in a bit. Let's keep exploring, and instead look at a make_debug that accepts a &String:
fn make_debug(_: &String) -> impl std::fmt::Debug {
42u8
}
What's the lifetime of that reference? Thanks to lifetime elision, we don't have to state it explicitly. But the implied lifetime is within the lifetime of the function itself. In other words, our borrow of the String expires completely when our function exits. We can prove that point a bit more by trying to return the reference:
fn make_debug(message: &String) -> impl std::fmt::Debug {
//42u8
message
}
The error message we get is a bit surprising, but quite useful:
error: cannot infer an appropriate lifetime
--> src/main.rs:4:5
|
2 | fn make_debug(message: &String) -> impl std::fmt::Debug {
| -------------------- this return type evaluates to the `'static` lifetime...
3 | //42u8
4 | message
| ^^^^^^^ ...but this borrow...
|
note: ...can't outlive the anonymous lifetime #1 defined on the function body at 2:1
--> src/main.rs:2:1
|
2 | / fn make_debug(message: &String) -> impl std::fmt::Debug {
3 | | //42u8
4 | | message
5 | | }
| |_^
help: you can add a constraint to the return type to make it last less than `'static` and match the anonymous lifetime #1 defined on the function body at 2:1
|
2 | fn make_debug(message: &String) -> impl std::fmt::Debug + '_ {
| ^^^^^^^^^^^^^^^^^^^^^^^^^
What's happening is we have essentially two lifetimes in our signature. The implied lifetime for message is the lifetime of the function, whereas the lifetime for impl Debug is 'static, meaning it either borrows no data or only borrows values that last the entire program (such as a string literal). We can even try to follow through with the recommendation and add some explicit lifetimes:
fn make_debug<'a>(message: &'a String) -> impl std::fmt::Debug + 'a {
message
}
fn test() -> impl std::fmt::Debug {
let value = "value".to_string();
make_debug(&value)
}
While this fixes make_debug itself, we can no longer call make_debug successfully from test:
error[E0597]: `value` does not live long enough
--> src/main.rs:11:16
|
7 | fn test() -> impl std::fmt::Debug {
| -------------------- opaque type requires that `value` is borrowed for `'static`
...
11 | make_debug(&value)
| ^^^^^^ borrowed value does not live long enough
12 | }
| - `value` dropped here while still borrowed
In other words, our return value from test() is supposed to outlive test itself, but value does not outlive test.
Challenge question Make sure you can explain to yourself (or a rubber duck): why did returning message work when we were passing by value but not by reference?
For the concrete type versions of make_debug, we essentially have a two-by-two matrix: whether we pass by value or reference, and whether we return the provided parameter or a dummy 42u8 value. Let's get this clearly recorded:
|
By value |
By reference |
| Use message |
Success: parameter owned by return value |
Failure: return value outlives reference |
Use dummy 42 |
Success: return value doesn't need parameter |
Success: return value doesn't need reference |
Hopefully the story with concrete types just described makes sense. But that leaves us with the question...
Why does polymorphism break things?
We see in the bottom row that, when returning the dummy 42 value, we're safe with both pass-by-value and pass-by-reference, since the returned value doesn't need the parameter at all. But for some reason, when we use a parameter T instead of String or &String, we get an error message. Let's refresh our memory a bit with the code:
fn make_debug<T>(_: T) -> impl std::fmt::Debug {
42u8
}
fn test() -> impl std::fmt::Debug {
let value = "value".to_string();
make_debug(&value)
}
And the error message:
error[E0597]: `value` does not live long enough
--> src/main.rs:10:16
|
6 | fn test() -> impl std::fmt::Debug {
| -------------------- opaque type requires that `value` is borrowed for `'static`
...
10 | make_debug(&value)
| ^^^^^^ borrowed value does not live long enough
11 | }
| - `value` dropped here while still borrowed
From within make_debug, we can readily see that the parameter is ignored. However, and this is the important bit: the function signature of make_debug doesn't tell us that explicitly! Instead, here's what we know:
make_debug takes a parameter of type TT may contain references with non-static lifetimes, we really don't know (also: very important!)- The return value is an
impl Debug
- We don't know what concrete type this return value has, but it must have a
'static lifetime
- The
impl Debug may rely upon data inside the T parameter
The outcome of this is: if T has any references, then their lifetime must be at least as large as the lifetime of the return impl Debug, which would mean it must be a 'static lifetime. Which sure enough is the error message we get:
opaque type requires that `value` is borrowed for `'static`
Notice that this occurs at the call to make_debug, not inside make_debug. Our make_debug function is perfectly valid as-is, it simply has an implied lifetime. We can be more explicit with:
fn make_debug<T: 'static>(_: T) -> impl std::fmt::Debug + 'static
Why the workarounds work
We previously fixed the compilation failure by making the type of the parameter concrete. There are two relatively easy ways to work around the compilation failure and still keep the type polymorphic. They are:
- Change the parameter from
_: T to _: &T
- Change the call site from
make_debug(&value) to make_debug(value)
Challenge Before reading the explanations below, try to figure out for yourself what these changes fix the compilation based on what we've explained so far.
Change parameter to &T
Our implicit requirement of T is that any references it contains have a static lifetime. This is because we cannot see from the type signature whether the impl Debug is holding onto data inside T. However, by making the parameter itself a reference, we change the ballgame completely. Suddenly, instead of just a single implied lifetime of 'static on T, we have two implied lifetimes:
- The lifetime of the reference, which we'll call
'a
- The lifetime of the
T value and the impl Debug, which are both still 'static
More explicitly:
fn make_debug<'a, T: 'static>(_: &'a T) -> impl std::fmt::Debug + 'static
While we cannot see from this type signature whether the impl Debug depends on data inside the T, we do know—by the definition of the impl Trait feature itself—that it does not depend on the 'a lifetime. Therefore, the only requirement for the reference is that it live as long as the call to make_debug itself, which is in fact true.
Change call to pass-by-value
If, on the other hand, we keep the parameter as T (instead of &T), we can fix the compilation issue by passing by value with make_debug(value) (instead of make_debug(&value)). This is because the requirement of the T value passed in is that it have 'static lifetime, and values without reference do have such a lifetime (since they are owned by the function). More intuitively: make_debug takes ownership of the T, and if the impl Debug uses that T, it will take ownership of it away from make_debug. Otherwise, when we leave make_debug, the T will be dropped.
Review by table
To sum up the polymorphic case, let's break out another table, this time comparing whether the parameter is T or &T, and whether the call is make_debug(value) or make_debug(&value):
|
Parameter is T |
Parameter is &T |
make_debug(value) |
Success: lifetime of the String is 'static |
Type error: passing a String when a reference expected |
make_debug(&value) |
Lifetime error: &String doesn't have lifetime 'static |
Success: lifetime of the reference is 'a |
Conclusion
Personally I found this behavior of impl Trait initially confusing. However, walking through the steps above helped me understand ownership in this context a bit better. impl Trait is a great feature in the Rust language. However, there may be some cases where we need to be more explicit about the lifetimes of values, and then reverting to the original big type signature approach may be warranted. Hopefully those cases are few and far between. And often, an explicit clone—while inefficient—can save a lot of work.
Learn more
Read more information on Rust at FP Complete and see our other learning material. If you're interested in getting help with your projects, check out our consulting and training offerings.
FP Complete specializes in server side software, with expertise in Rust, Haskell, and DevOps. If you're interested in learning about how we can help your team succeed, please reach out for a free consultation with one of our engineers.