At PMG we maintain a custom terraform provider that talks to a few of our internal, platform APIs. APIs can change and evolve, and without a layer in between the API responses and terraform data sourcess, a custom TF provider can break when the API changes.
I know this because I naïvely did this exact thing: got a JSON response in a []map[string]interface{} and set it as data on a terraform data source. Then the API changed in a backwards campatible way (added fields), but the terraform provider broke and started erroring out.
Here’s an example of what not to do:
package example
import (
"context"
"encoding/json"
"github.com/hashicorp/terraform-plugin-sdk/v2/diag"
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
"net/http"
"strings"
)
func dataSourceApiThings() *schema.Resource {
return &schema.Resource{
ReadContext: dataSourceApiThingsRead,
Schema: map[string]*schema.Schema{
"things": &schema.Schema{
Type: schema.TypeList,
Computed: true,
Elem: &schema.Resource{
Schema: map[string]*schema.Schema{
"id": &schema.Schema{
Type: schema.TypeString,
Computed: true,
},
"name": &schema.Schema{
Type: schema.TypeString,
Computed: true,
},
"enabled": &schema.Schema{
Type: schema.TypeBool,
Computed: true,
},
},
},
},
},
}
}
func dataSourceApiThingsRead(ctx context.Context, d *schema.ResourceData) diag.Diagnostics {
var diags diag.Diagnostics
resp, err := http.Get("http://example.com/things")
if err != nil {
return diag.FromErr(err)
}
defer resp.Body.Close()
// don't do this! any time the API changes the provider will break
// with errors like "can't set field with this path..."
things := make([]map[string]interface{}, 0)
err = json.NewDecoder(resp.Body).Decode(&things)
if err != nil {
return diag.FromErr(err)
}
thingIds := make([]string, len(things))
for i, thing := range things {
thingIds[i] = thing["id"].(string)
}
if err := d.Set("things", things); err != nil {
return diag.FromErr(err)
}
d.SetId(strings.Join(thingIds, ","))
return diags
}
I should have decoded my JSON into a custom struct that acted as a layer between the API responses and the terraform data source results, but this can cause terraform to complain if the struct’s fields don’t match the data source’s schema — and you may not want to have TitleCase terraform data source attributes.
One option is to decode API responses into a struct, then convert the struct back to a map for Terraform.
package example
type ApiThing struct {
Id string `json:"id"`
Name string `json:"name"`
Enabled bool `json:"enabled"`
}
func (a ApiThing) ToMap() map[string]interface{} {
return map[string]interface{}{
"id": a.Id,
"name": a.Name,
"enabled": a.Enabled,
}
}
The data source read function now decodes the API response to a []ApiThing and instead of setting the data source value from the API response directly, we use ToMap to get the data for terraform to return in the datasource. This insulates changes in the API from the terraform provider.
things := make([]ApiThing, 0)
err = json.NewDecoder(resp.Body).Decode(&things)
// and the code that converts it
thingIds := make([]string, len(things))
thingMaps = make([]map[string]interface{}, len(things))
for i, thing := range things {
thingIds[i] = thing.Id
thingMaps[i] = thing.ToMap()
}
if err := d.Set("things", things); err != nil {
return diag.FromErr(err)
}
Digging a little deeper, I was hoping for there to be a way to use struct tags so no ToMap function would be required. There is! Terraform makes use of this mapstructure library, which allows you to add mapstructure:"name" tags to map the field names to map names. No more ToMap function required.
type ApiThing struct {
Id string `json:"id" mapstructure:"id"`
Name string `json:"name" mapstructure:"name"`
Enabled bool `json:"enabled" mapstructure:"enabled"`
}
And in the data source read function, you use the decoded API response directly.
things := make([]ApiThing, 0)
err = json.NewDecoder(resp.Body).Decode(&things)
// ...
thingIds := make([]string, len(things))
for i, thing := range things {
thingIds[i] = thing.Id
}
if err := d.Set("things", things); err != nil {
return diag.FromErr(err)
}
A possible downside here is that mapstructure will not omit fields. So if hiding a field is desired, the only option is to not include it in the custom struct. Additionally, there’s no sense of context in which mapstructure is decoding. A response from a write endpoint may share the same sturcture as a read, but have more values returned in that write context. Something like that would require a method similar to ToMap.
Christopher Davis 