How code generation wrote our API and CLI by Mat Ryer

We recently released the Pace API, along with an accompanying command line tool, and various client libraries, all of which enable programmatic integration with Pace.

We use Oto to describe our RPC (Remote Procedure Call) API using Go interfaces and structs, and a variety of templates to generate the code we need.

All methods in the API use a input and output struct for the request and response, supporting all JSON data types; strings, numbers, bools, objects and arrays.

Describing the API

We describe our API using Go interfaces and structs.

Here’s a sample from our CardsService that describes the GetCard method:

package pace

// CardsService allows you to programmatically manage cards in Pace.
type CardsService interface {
	// GetCard gets a card by ID.
	GetCard(GetCardRequest) GetCardResponse
}

// GetCardRequest is the input object for GetCard.
type GetCardRequest struct {
	// OrgID is the ID of the org.
	// example: "your-org-id"
	OrgID string
	// CardID is the ID of the card to get.
	// example: "123"
	CardID string
}

// GetCardResponse is the output object for GetCard.
type GetCardResponse struct {
	// Card is the card.
	Card Card
}

type Card struct {
	ID           string
	CTime        string
	Title        string
	Body         string
	BodyHTML     string
	// simplified to save your eyes
}

Since this is real Go code, our IDEs build and lint this along with the rest of our code.

How Oto parses the definition

Oto uses the golang.org/x/tools/go/packages package (among others from the standard library) to understand the Go code, and turn the definition files into a workable data structure.

The following types are taken from the parser inside Oto:

// Service describes a service, akin to an interface in Go.
type Service struct {
	Name    string   `json:"name"`
	Methods []Method `json:"methods"`
	Comment string   `json:"comment"`
}

// Method describes a method that a Service can perform.
type Method struct {
	Name           string    `json:"name"`
	NameLowerCamel string    `json:"nameLowerCamel"`
	InputObject    FieldType `json:"inputObject"`
	OutputObject   FieldType `json:"outputObject"`
	Comment        string    `json:"comment"`
}

// etc

Now that we have Go data that describes the interfaces, methods and structs, we can use templates to generate code.

Oto uses Plush templates from the Buffalo project

Code generation

Oto is essentially a code generation tool, because most of the work needed to wire up an implementation to an endpoint is predictable boilerplate that we would prefer not to write.

When generics lands in Go, we will be able to reduce the amount of boilerplate code generated in favour of generic methods. In this world, we may not need to generate much plumbing code at all.

Generating server-side plumbing

The first thing we generate is the server-side plumbing for the implementation of the service.

The template that was used to generate this code is available as part of otohttp

package api

// CardsService allows you to programmatically manage cards in Pace.
type CardsService interface {
	// GetCard gets a card by ID.
	GetCard(context.Context, GetCardRequest) (*GetCardResponse, error)
}

type cardsServiceServer struct {
	server       *otohttp.Server
	cardsService CardsService
}

// RegisterCardsService registers the CardsService implementation with the otohttp Server.
func RegisterCardsService(server *otohttp.Server, cardsService CardsService) {
	handler := &cardsServiceServer{
		server:       server,
		cardsService: cardsService,
	}
	server.Register("CardsService", "GetCard", handler.handleCardsServiceGetCard)
}

func (s *cardsServiceServer) handleCardsServiceGetCard(w http.ResponseWriter, r *http.Request) {
	var request GetCardRequest
	if err := otohttp.Decode(r, &request); err != nil {
		s.server.OnErr(w, r, fmt.Errorf("CardsService GetCard %s", err))
		return
	}
	response, err := s.cardsService.GetCard(r.Context(), request)
	if err != nil {
		s.server.OnErr(w, r, fmt.Errorf("CardsService GetCard %s", err))
		return
	}
	if err := otohttp.Encode(w, r, http.StatusOK, response); err != nil {
		s.server.OnErr(w, r, fmt.Errorf("CardsService GetCard %s", err))
		return
	}
}

In the above code, we generate:

A new CardsService Go interface that describes the methods of the service. Notice that context.Context and error types were added to the signature of the methods
The cardsServiceServer struct couples the otohttp.Server with the service implementation, and has real http.Handler methods like handleCardsServiceGetCard that handles the RPC calls
The RegisterCardsService method uses the Register method on the otohttp.Server to bind the route CardsService.GetCard to the appropriate handler

All we have to do is write a struct that implements the new CardsService interface, and the generated code, along with the code inside otohttp, does the rest for us.

Generating clients

A client is used to access the services on the remote server.

They aren’t always necessary if you’re working with plain-old JSON/HTTP but due to the security measures in Pace, we need to securely sign each request with a secret cryptographic key so we can be sure it is being made by a trusted partner. This can be tricky, so we generate clients that do it for you.

The first client we generate is a Go package, which we use in our tests.

By dog-fooding the generated Go client in our tests, we can not only ensure the services are functioning as we expect them to, but also that the client itself works.

The generated code for the GetCard client (generated from a template file in the github.com/pacedotdev/pace project) looks like this:

// GetCard gets a card.
func (s *CardsService) GetCard(ctx context.Context, r GetCardRequest) (*GetCardResponse, error) {
	requestBodyBytes, err := json.Marshal(r)
	if err != nil {
		return nil, errors.Wrap(err, "CardsService.GetCard: marshal GetCardRequest")
	}
	signature, err := generateSignature(requestBodyBytes, s.client.secret)
	if err != nil {
		return nil, errors.Wrap(err, "CardsService.GetCard: generate signature GetCardRequest")
	}
	url := s.client.RemoteHost + "/api/CardsService.GetCard"
	s.client.Debug(fmt.Sprintf("POST %s", url))
	s.client.Debug(fmt.Sprintf(">> %s", string(requestBodyBytes)))
	req, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(requestBodyBytes))
	if err != nil {
		return nil, errors.Wrap(err, "CardsService.GetCard: NewRequest")
	}
	req.Header.Set("X-API-KEY", s.client.apiKey)
	req.Header.Set("X-API-SIGNATURE", signature)
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("Accept-Encoding", "gzip")
	req = req.WithContext(ctx)
	resp, err := s.client.HTTPClient.Do(req)
	if err != nil {
		return nil, errors.Wrap(err, "CardsService.GetCard")
	}
	defer resp.Body.Close()
	var response struct {
		GetCardResponse
		Error string
	}
	var bodyReader io.Reader = resp.Body
	if strings.Contains(resp.Header.Get("Content-Encoding"), "gzip") {
		decodedBody, err := gzip.NewReader(resp.Body)
		if err != nil {
			return nil, errors.Wrap(err, "CardsService.GetCard: new gzip reader")
		}
		defer decodedBody.Close()
		bodyReader = decodedBody
	}
	respBodyBytes, err := ioutil.ReadAll(bodyReader)
	if err != nil {
		return nil, errors.Wrap(err, "CardsService.GetCard: read response body")
	}
	if err := json.Unmarshal(respBodyBytes, &response); err != nil {
		if resp.StatusCode != http.StatusOK {
			return nil, errors.Errorf("CardsService.GetCard: (%d) %v", resp.StatusCode, string(respBodyBytes))
		}
		return nil, err
	}
	if response.Error != "" {
		return nil, errors.New(response.Error)
	}
	return &response.GetCardResponse, nil
}

Notice that this code explicitly mentions the input and output objects GetCardRequest and GetCardResponse - this is an example of code that would be replaced when Go gets generics.

JavaScript/TypeScript clients

It isn’t just Go code we can generate, in fact, the templates don’t really know what they’re producing.

We generate a TypeScript client from a slightly modified version of the TypeScript template in Oto, that yields something like this for the GetCard method:

// GetCard gets a card by ID.
async getCard(getCardRequest: GetCardRequest = null) {
	if (getCardRequest == null) {
		getCardRequest = new GetCardRequest();
	}
	const headers: HeadersInit = new Headers();
	headers.set('Accept', 'application/json');
	headers.set('Content-Type', 'application/json');
	await this.client.headers(headers);
	const response = await fetch(this.client.basepath + 'CardsService.GetCard', {
		method: 'POST',
		headers: headers,
		body: JSON.stringify(getCardRequest),
	})
	return response.json().then((json) => {
		if (json.Error) {
			throw new Error(json.Error);
		}
		return new GetCardResponse(json);
	})
}

Documentation

Our documentation is a Svelte JS app, available to browse at https://pace.dev/docs/api.

We generate the .svelte components that make up the documentation front-end.

The comments extracted from the original definition files are preserved throughout, and we also use them in the docs.

Extending our API

If we want to make changes to our clients, we only have to modify the tempalte and regenerate the code. We do not have to make the same changes over and over again for every method.

Similarily, if we want to add a new service or method, we only have to:

Update the definition package code
Use the oto command to generate server stubs, and client code
The build fails because the implementation struct no longer matches the generated Go interface (since we added new methods). So we implement the methods to satisfy the interface

Frequently asked questions

A few questions come up again and again, so we’ve answered some of them here.

Why not use gRPC?

gRPC has some benefits over our approach, and also some key disadvantages which ultimately pushed us to build Oto.

gRPC uses a binary protocol, which results in smaller data payload sizes when compared to text-based JSON. In systems with lots of messages flying around, this can make a big difference but isn’t (yet?) a high priority for us due to the nature of the app we’re building. Instead, API discoverability and developer friendliness are more important.

The primary disadvantage we encountered was that the gRPC tooling requires you to open a port for the binary comms, which was not possible to do on Google App Engine (standard environment), where Pace is deployed.

Secondly, rather than obfuscate the message data (humans struggle reading binary) we wanted a more user friendly JSON API which was more familiar to developers. There are a range of great packages that allow you to expose JSON services alongside the binary gRPC ones, but they work by proxying to the binary port, rather than providing a standalone solution within themselves.

So in our case, developer comfortability and familiarity (ours and our future API consumers) is more important than most of the technical arguments that you might make in favour of gRPC.

Oto uses Go interfaces to describe the API, and in all honesty, we generally try to use Go for as much as we can.

Finally, it’s worth mentioning that Oto has a JSON/HTTP implementation (called otohttp) but that this isn’t the only possible implementation. For example, it would be fairly trivial to add binary support to Oto.

Learn more about what we're doing at Pace.

A lot of our blog posts come out of the technical work behind a project we're working on called Pace.

We were frustrated by communication and project management tools that interrupt your flow and overly complicated workflows turn simple tasks, hard. So we decided to build Pace.

Pace is a new minimalist project management tool for tech teams. We promote asynchronous communication by default, while allowing for those times when you really need to chat.

We shift the way work is assigned by allowing only self-assignment, creating a more empowered team and protecting the attention and focus of devs.

We're currently live and would love you to try it and share your opinions on what project management tools should and shouldn't do.

What next? Start your 14 day free trial to see if Pace is right for your team

How code generation wrote our API and CLI.