Blogg
Här finns tekniska artiklar, presentationer och nyheter om arkitektur och systemutveckling. Håll dig uppdaterad, följ oss på Twitter
Här finns tekniska artiklar, presentationer och nyheter om arkitektur och systemutveckling. Håll dig uppdaterad, följ oss på Twitter
In this blog post, we’ll use Golang to boot and test a microservice environment.
Sometimes testing a microservice environment presents challenges not typically encountered when developing more traditional applications of the monolith kind. For example, our modern microservice environment may consist of a number of containers (we’re using Docker) each providing a service or a support function for managing the microservice landscape. In this blog post, I’ll be using the tutorial series my colleague Magnus has published on this blog and on github as our sample Microservice landscape. It consists of one public-facing service backed by several internal microservices and no less than six supporting services:
Want to try/run this yourself? Clone the source code and follow instructions in the blog post!
Part of the challenge here is to make all these components work together, something that requires a non-trivial amount of configuration. Also, making sure things stays that way over time is equally important. Typically, you have a CI/CD pipeline to detect regressions, but with container technology it is quite easy to bootstrap all components on your developer laptop (given enough RAM). This blog post won’t go into details about the inner workings of the sample project, what I intend to show is how we can use a simple golang program to efficiently bootstrap the entire landscape using Docker compose, wait for service availability, run a number of endpoint tests and finally shut it all down again. In other words - a very simplistic testing framework for early detection of configuration or service regressions. To accomplish this we’ll use my new favorite general-purpose non–JVM language - Golang!
This program was actually inspired by a bash shell script and it is certainly just as easy to do this using whatever console-friendly language you prefer. I like Go due to its relative simplicity, quick compile time, goroutines, portability and rich set of libraries.
It’s actually a rather trivial task, divided into a number of distinct steps:
Let’s start by having a look at a sample YAML file for specifying the environment we want to boot and test:
---
title: Microservices sample test file #(1)
docker_compose_root: /Users/myuser/projects/microservices-workshop/solution-7 #(2)
docker_compose_file: docker-compose.yml #(3)
services: #(4)
- http://192.168.99.100:8761
- http://192.168.99.100:8761/eureka/apps/edgeserver
- http://192.168.99.100:8761/eureka/apps/product
- http://192.168.99.100:8761/eureka/apps/productapi
- http://192.168.99.100:8761/eureka/apps/productcomposite
- http://192.168.99.100:8761/eureka/apps/recommendation
- http://192.168.99.100:8761/eureka/apps/review
oauth: #(5)
url: https://192.168.99.100:9999/uaa/oauth/token
client_id: acme
client_password: acmesecret
scope: webshop
grant_type: password
username: user
password: password
token_key: access_token
endpoints: #(6)
- url: https://192.168.99.100/api/product/1046
auth_method: TOKEN
- url: https://192.168.99.100/api/product/1337
auth_method: TOKEN
- url: https://192.168.99.100/api/product/7331
auth_method: TOKEN
The YAML files defines some environment specifics such as title, docker project root directory and file, what microservices to query Eureka for before fetching OAuth token and testing services.
So, how does our little golang program tie this together? I would recommend starting by taking a look directly at the main.go source file main() method:
func main() {
CallClear()
// 1. Check required command-line tools are present
// docker-compose etc.
CheckDocker()
// 2. Load service specification from yaml
t := LoadSpecification()
// 3. Start using docker-compose up -d, then use defer to make sure it's torn down afterwards.
DockerComposeUp(t)
defer DockerComposeDown(t)
// 4. Then wait for specified microservices
awaitServicesHasStarted(t)
// 5. When all are started, get and store OAuth token
StoreOAuthToken(t)
// 6. execute list of endpoint HTTP calls. (blocks until finished)
runEndpoints(t)
}
CallClear() performs a “clear” or “cls” on the executing console depending on the host OS. For the sake of clarity, I’ve omitted some console output using fancy VT100 console cursor control, will return to that later.
exec.LookPath is a convenience method for checking if a certain executable is present on the PATH, in this case ‘docker-compose’. If not present, we use the log.Fatal() to log and exit.
func CheckDocker() {
_, err := exec.LookPath("docker-compose")
if err != nil {
log.Fatal("docker-compose not installed, fix!")
}
fmt.Printf("docker-compose installed OK\n")
}
func LoadSpecification() (TestDef) {
spec := parseSpecFile()
dir, _ := os.Getwd()
dat, _ := ioutil.ReadFile(dir + "/" + spec)
var t TestDef
yaml.Unmarshal([]byte(dat), &t)
fmt.Println("Loaded specification '" + t.Title + "'")
return t
}
Not too verbose - get the .yaml file cmd-line arg, current directory, read the file and pass a reference to an uninitialized TestDef struct into the yaml.Unmarshal method. The yaml library does all the parsing for us, allowing us to return the populated struct to the main() method.
func DockerComposeUp(t TestDef) {
cmd := exec.Command("docker-compose", "-f", t.DockerComposeFile, "up", "-d")
cmd.Dir = t.DockerComposeRoot
env := os.Environ()
env = append(env, fmt.Sprintf("PROJECT_ROOT=%s", t.DockerComposeRoot))
cmd.Env = env
cmd.Run()
fmt.Println("Docker starting up using " + t.DockerComposeRoot + "/" + t.DockerComposeFile + " ...")
}
The exec.Command prepares a Cmd using varargs separated arguments to the docker-compose executable. ‘-f’ specifies the file name and ‘-d’ tells it to run in a daemon mode. Eventually, we’ll call Run() on the cmd instance.
Note how we specify the docker-compose file using a value from our testDef and how we append a PROJECT_ROOT environment variable to the env of the Cmd. Each Cmd is only active for a given Cmd so it’s not possible in go to first do this:
// WON'T WORK
cmd1 := exec.Command("export", "PROJECT_ROOT=" + t.DockerComposeRoot)
cmd1.Run()
cmd2 := exec.Command("docker-compose", "-f", t.DockerComposeFile, "up", "-d")
cmd2.Run()
// WON'T WORK!!
In the faulty snippet above, the ‘export’ won’t carry over to cmd2, making use of such a environment variable in the docker-compose.yml yield nil, hence the use of the environment trick in the correct example further up.
How do we know when all infrastructure and services have started and is ready for testing? Our best bet is to utilize Eureka - remember these?
services: #(4)
- http://192.168.99.100:8761
- http://192.168.99.100:8761/eureka/apps/config-server
...
- http://192.168.99.100:8761/eureka/apps/product
The first URL is simply the root of the Eureka service - if Eureka isn’t up we can’t check the state of services that has registered with the discover service. Under http://192.168.99.100:8761/eureka/apps/[service name] Eureka publishes status pages (XML) for each registered service. For our purpose, we’re OK as long as a “OK” HTTP status code is returned.
func awaitServicesHasStarted(t TestDef) {
wg := sync.WaitGroup{}
fmt.Println("Waiting for all microservices to start...")
consoleRow++
fmt.Println("")
consoleRow++
for _, service := range t.Services {
wg.Add(1)
consoleRow++
go PollService(service, &wg, len(t.Services), consoleRow)
}
wg.Wait() // Blocks here until all wg.Done() has been called within PollService.
// Fix cursor position after all services have started
consoleRow+=3
fmt.Printf("\033[%d;0H", consoleRow) // Move cursor to row
}
Here we see some fun golang features in action, namely the WaitGroup and the go keyword. What this snippet does is that it will spawn t.Services number of goroutines executing the PollService method. For each “go” wg.Add(1) tells the waitgroup to increment the number of future wg.Done() it needs to receive until continuing processing after wg.Wait().
func PollService(service string, wg *sync.WaitGroup, total int, consoleRow int) {
Cprint(consoleRow, 0, service)
Cprint(consoleRow, 60, "... waiting ")
req, _ := http.NewRequest("GET", service, nil)
var DefaultTransport http.RoundTripper = &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
for {
resp, err := DefaultTransport.RoundTrip(req)
if err != nil || resp.StatusCode > 299 {
Cprint(consoleRow, 0, service)
Cprint(consoleRow, 60, "... waiting ")
} else {
Cprint(consoleRow, 0, service)
Cprint(consoleRow, 60, "done ")
wg.Done()
return
}
time.Sleep(time.Second * 1)
}
}
Ignore all the Cprint calls for now, they’re just for the VT100-style console output. The important part is the HTTP GET requests performed using net/http’s DefaultTransport. DefaultTransport? Why not use the standard net/http client golang provides? Well - the easiest way of having golang’s net/http client to accept servers with self-signed certificates is by setting the TLSClientConfig seen in the code.
Apart from the cert fix (and the ugly Cprint statements), the program will keep looping issuing one request per second to whatever Url that was specified in the service parameter until a < 299 HTTP status is returned.
For example, http://192.168.99.100:8761 - which in our microservice landscape is the Eureka start page - won’t return 200 OK until the Eureka service is up and running. Somewhat later, the “Product service” status can be queried by asking Eureka it’s state at the http://192.168.99.100:8761/eureka/apps/product. So remember, it is not the actual service endpoint we are checking, we are just making sure it has booted and registered itself with its discovery service.
Actual endpoint testing is probably much more complex in a real-world scenario than here - you’d want full support for other HTTP methods, POST bodies, headers, response parsing etc. - which is out of scope for this little exercise. The service testing code (e.g. runEndpoints) is actually very similar to the PollService one, with the difference that we’re adding a HTTP header with whatever OAuth token our request to StoreOAuthToken function yielded:
req := BuildHttpRequest(endpoint.Url, endpoint.Method)
var DefaultTransport http.RoundTripper = &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
if endpoint.Auth_method == "TOKEN" {
req.Header.Add("Authorization", "Bearer " + TOKEN)
}
We then try each service up to 10 times before giving up:
retries := 10
for i := 0; i < retries; i++ {
resp, err := DefaultTransport.RoundTrip(req)
if err != nil || resp.StatusCode > 299 {
Cprint(consoleRow, 0, endpoint.Url)
Cprint(consoleRow, 70, "... failed. Retrying " + strconv.Itoa(i) + "/" + strconv.Itoa(retries) + " ... ")
} else {
Cprint(consoleRow, 0, endpoint.Url)
Cprint(consoleRow, 70, "... OK ")
wg2.Done()
break
}
time.Sleep(time.Second * 3)
if (i == retries - 1) {
wg2.Done()
Cprint(consoleRow, 0, endpoint.Url)
Cprint(consoleRow, 70, "... All attempts failed, something is broken. ")
}
}
Again, we use a WaitGroup to make sure each goroutine can finish before the program exits.
Finally, the little CPrint method used to output program state to the console window:
var l sync.Mutex
func Cprint(row int, col int, text string) {
l.Lock()
fmt.Printf("\033[%d;%dH", row, col)
fmt.Print(text)
l.Unlock()
}
We’re simply using Printf with the VT100 escape sequence for positioning the cursor and row/col and then printing text. Since each goroutine may call this code to update console output at any time we need to use a lock mechanism or the output will become really garbled. Alternatively, we could have used a sync go channel instead, see bottom of this blog post for an example.
Below, we have a sample output from a run.
> go run src/github.com/eriklupander/mstest/*.go spec.yml
Starting up...
docker-compose installed OK
Loaded specification 'Microservices sample test file'
Docker starting up using /Users/eriklupander/privat/blog-microservices/docker-compose.yml ...
Waiting for all microservices to start...
http://192.168.99.100:8761 done
http://192.168.99.100:8761/eureka/apps/edgeserver done
http://192.168.99.100:8761/eureka/apps/product done
http://192.168.99.100:8761/eureka/apps/productapi done
http://192.168.99.100:8761/eureka/apps/productcomposite done
http://192.168.99.100:8761/eureka/apps/recommendation done
http://192.168.99.100:8761/eureka/apps/review done
Getting OAuth token ... OK
https://192.168.99.100/api/product/1046 ... OK
https://192.168.99.100/api/product/1337 ... OK
https://192.168.99.100/api/product/7331 ... OK
All done.
Docker shutting down...
Bonus snippet of code using Go channels for handling (synchronized) console printing without explicit use of mutexes and/or locks.
var consoleChannel chan CText // Declare privately scoped channel that we can pass console print
// statements to
func init() {
// Maybe some other stuff
go pollConsoleChannel() // Start a goroutine that will handle receives on the channel
}
func pollConsoleChannel() {
consoleChannel = make(chan CText) // Instantiate unbuffered (e.g. sync) channel
// Will iterate until main program exits, blocks at <-.
for {
msg := <- consoleChannel // Blocks here until a message is received.
fmt.Printf("\033[%d;%dH", msg.Row, msg.Col) // Position cursor
fmt.Print(msg.Text) // Prints text at cursor position
}
}
/* Application invokes this func to print to console */
func ChPrint(row int, col int, text string) {
msg := CText{ // Build a simple instance of the CText struct
row,
col,
text,
}
consoleChannel <- msg // Submit message on channel
}
/* Simple struct for encapsulating row, col and text to log */
type CText struct {
Row int
Col int
Text string
}