Ergonomic REST APIs with Dynamic Object Expansion

Teddy LiuCo-Founder @ Pocketflows

A box with arrows surrounding it, showing its expansion

Premier API companies, like Stripe, implement an additional expand parameter on all their endpoints. Generally, nested objects in the Stripe API are not expanded by default and instead are represented by their id . This parameter gives you ability to inflate the nested objects returned from the API and get their attributes and also recursively inflate the nested objects within them.

At Pocketflows, we’re building embedded APIs to help VSaaS and enterprise companies build high-quality marketing tools faster and better. It’s our goal to offer the best API experience, so we have implemented the expand parameter and wanted to share our thoughts.

Motivation

Why have an expand parameter at all? We can see its utility by examining alternative approaches. Note that GraphQL actually solves many of the following concerns, but that’s a topic for a future blog post.

Approach 1: Expand Everything

One option (that Pocketflows implemented in its MVP) is to expand every nested object fully and recursively. For example, in Pocketflows, there is a Member which associates both a User and a Rewards Program and gives . If we take the “expand everything” approach, a GET to /members/mem_1 would return

{
  "id": "mem_1",
  "point_total": 1000,
  "point_balance": 2000,
  "user": {
    "id": "u_1",
    "phone_number": "+1234567890"
  },
  "rewards_program": {
    "id": "rp_1",
    "points_earned_per_dollar_spent": 2,
    "point_redemption_value": "0.01",
    "points_for_member_class_attendance": 100
  }
}

Pros

• Simplifies serialization on the server — just return all fields and do so recursively
• The API caller will never have to specify additional parameters and will always get the data they’re looking for.

Cons

• Likely sends way more data than the API caller needs and this problem only worsens as the number of resources and amount of nesting in your API grows
• With scaled API usage, this approach will consume a lot of bandwidth unnecessarily. You will be sending bytes for nested objects that are never used. This is particularly exacerbated when requesting a list of objects. For example, getting all Members would return an array of the above object and for many of the Members, rewards_programs would be exactly the same object!

One final subtlety is that with this approach, you need to be careful about implicit cycles within your object model. If there was a one-to-one relationship between two objects and we expand everything naively, we end up with a situation similar to infinite recursion!

Approach 2: Expand Nothing

On the opposite extreme, we can choose to expand none of the nested objects. Keeping the example from above, with Members, making a GET request to /members/mem_1 would return:

{
  "id": "mem_1",
  "point_total": 1000,
  "point_balance": 2000,
  "user": "u_1",
  "rewards_program": "rp_1"
}

This approach addresses some of the concerns from the “expand everything” approach.

Pros

• There are no risks of infinite recursion in serialization.
• Depending on the underlying DB data model, it may not be necessary to even query multiple tables.
• The amount of data sent is minimal, preserving bandwidth.
• In the case of arrays, the amount of duplication is minimal — limited to just the IDs.

Cons

• If the API caller does want additional information about the User or Rewards Program , they need to make a separate API calls to /users/u_1 and /rewards_program/rp_1 .

Approach 3: expand parameter

The “expand nothing” approach has many benefits with one major limiting downside: when the API caller would prefer to have more information about a nested object, they need to make separate calls to the API. We address this by providing an opt-in parameter: expand. The default behavior of all endpoints is to “expand nothing” as above, but when API callers want to view nested objects in the response, they can provide the expand parameter and get the data in the same response.

Continuing with the example, suppose an API caller would like to view a Member’s point total and would also like their phone number to send them a notification. They can now include the expand parameter in the GET request: /members/:id?expand[]=user . This returns

{
  "id": "mem_1",
  "point_total": 1000,
  "point_balance": 2000,
  "user": {
    "id": "u_1",
    "phone_number": "+1234567890"
  },
  "rewards_program": "rp_1"
}

The result selectively expands the User nested object but leaves the Rewards Program unexpanded. This gives the API caller the requested information about the User (their phone number) while not giving redundant information about the Rewards Program . There are few tradeoffs here! We’re minimizing the amount of bandwidth and redundant information while still giving all the relevant information in one API call. The API caller only needs to provide an extra query parameter.

Semantics

Now that we have motivated the utility of the expand parameter, we should also clarify the way it works! expand is an array of strings passed to the query parameters of GET requests and the request body of POST and DELETE requests.

For GET requests, arrays are encoded with the expand[] scheme. For example, if the intent is to expand both user and rewards_program , the query parameters should be expand[]=user&expand[]=rewards_program. This will be interpreted as

expand = ["user", "rewards_program"]

by the API.

Each of the strings should correspond to keys in the response object that have expandable values. In a Member object:

{
  "id": "mem_1",
  "point_total": 1000,
  "point_balance": 2000,
  "user": "u_1",
  "rewards_program": "rp_1"
}

"user" and "rewards_program" are valid values.

In addition, recursively nested objects can also be expanded with a dot notation. For example, an Event in Pocketflows has an associated Member. If we would like to see the associated Member’s phone number, we would expand the "member" key and then the "user" key in a Member. The API call might be POST with

{
  ...rest,
  "expand": ["member.user"]
}

returning the result

{
  "id": "ev_1",
  "member": {
    "id": "mem_1",
    "user": {
      "id": "u_1",
      "phone_number": "+1234567890"
    },
    "rewards_program": "rp_1"
  },
  "event_name": "user.book"
}

Note that "member.user" automatically implies the expansion of "member" . It is not necessary to pass both "member" and "member.user" .

For endpoints that return multiple objects or expanding nested objects in arrays, the expand parameter needs to pass the next relevant key, skipping any potential array indexing. Again, the rule for matching keys must be followed. If you make a GET to /members, you receive the result

{
  "members": [
    {
      "id": "mem_1",
      "user": "u_1",
      ...
    },
    {
      "id": "mem_2",
      "user": "u_2",
      ...
    }
  ]
}

To expand the "user" key for all of these, pass ?expand[]=members.user to the query to receive:

{
  "members": [
    {
      "id": "mem_1",
      "user": { "id": "u_1", "phone_number": "+1234567890" },
      ...
    },
    {
      "id": "mem_2",
      "user": { "id": "u_2", "phone_number": "+0987654321" },
      ...
    }
  ]
}

Restrictions and Errors

To prevent excess querying that stems from requesting too many layers of nested objects, we can restrict the amount of chaining allowed by the expand parameter. Stripe chooses a value of 3; providing deeper nesting should result in an error. At the moment, Pocketflows makes no such restrictions! Arbitrary depth can be provided to our API with no errors.

If the API caller passes a key that’s not valid on the object, either because the key is misspelled or the key does not correspond to an expandable object, this should generate an error from the API.

Implementation

At Pocketflows, we’re building our API with Ruby on Rails. Check out part two to learn tips and tricks from our sample implementation!

Conclusion

The expand provides flexibility to API callers while providing sensible and minimal defaults for all endpoints — unless otherwise specified, the amount of data sent is minimal, conserving bandwidth and database utilization. Implementing the expand parameter is relatively straightforward, and we hope you also include it in your API!

Are you looking to build marketing tools in your VSaaS or consumer-facing business? At Pocketflows, we're creating embedded APIs to build high-quality marketing products. We're incorporating the best API practices and creating the best developer experience. If you're interested in learning more, click the button below to schedule time with us!