Go - Web Application 7

The http.Server struct

Although http.ListenAndServe() is very useful in short examples and tutorials, in real-world applications it’s more common to manually create and use a http.Server struct instead. Doing this opens up the opportunity to customize the behavior of your server.

So in preparation for that, let’s quickly update our main.go file to stop using the http.ListenAndServe() shortcut, and manually create and use a http.Server struct instead.

func main() {
    addr := flag.String("addr", ":4000", "HTTP network address")
    dbDriver := flag.String("dbdriver", "mysql", "Database driver name")
    dsn := flag.String("dsn",
        "zeb:zebpwd@tcp(localhost:3306)/snippetbox?parseTime=true",
        "MySQL data source name")
    flag.Parse()

    logger := slog.New(slog.NewTextHandler(os.Stdout, nil))

    db, err := openDB(*dbDriver, *dsn)
    if err != nil {
        logger.Error(err.Error())
        os.Exit(1)
    }
    defer db.Close()

    templateCache, err := newTemplateCache()
    if err != nil {
        logger.Error(err.Error())
        os.Exit(1)
    }

    formDecoder := form.NewDecoder()

    sessionManager := scs.New()
    sessionManager.Store = mysqlstore.New(db)
    sessionManager.Lifetime = 12 * time.Hour

    app := &application{
        logger:         logger,
        snippet:        &models.SnippetModel{DB: db},
        templateCache:  templateCache,
        formDecoder:    formDecoder,
        sessionManager: sessionManager,
    }

    // Initialize a new http.Server struct. We set the Addr and Handler fields so that the server 
    // uses the same network address and routes as before.
    srv := &http.Server{
        Addr: *addr,
        Handler: app.routes(),
    }

    logger.Info("starting server", "addr", *addr)

    // Call the ListenAndServe() method on our new http.Server struct to start the server.
    err = srv.ListenAndServe()
    logger.Error(err.Error())
    os.Exit(1)
}

 

The server error log

It’s important to be aware that Go’s http.Server may write its own log entries relating to things like unrecovered panics, or problems accepting or writing to HTTP connections.

By default, it writes these entries using the standard logger — which means they will be written to the standard error stream (instead of standard out like our other log entries), and they won’t be in the same format as the rest of our nice structured log entries.

To demonstrate this, let’s add a deliberate error to our application and set a Content-Length header with an invalid value on our responses. Go ahead and update the render() helper like so:

func (app *application) render(w http.ResponseWriter, 
                               r *http.Request, 
                               status int, 
                               page string, 
                               data templateData) {
    ts, ok := app.templateCache[page]
    if !ok {
        err := fmt.Errorf("the template %s does not exist", page)
        app.serverError(w, r, err)
        return
    }

    buf := new(bytes.Buffer)

    err := ts.ExecuteTemplate(buf, "base", data)
    if err != nil {
        app.serverError(w, r, err)
        return
    }

    // Deliberate error: set a Content-Length header with an invalid (non-integer) value.
    w.Header().Set("Content-Length", "this isn't an integer!")

    w.WriteHeader(status)

    buf.WriteTo(w)
}

Then run the application, make a request to http://localhost:4000 , and check the application log. You should see that it looks similar to this:

zzh@ZZHPC:/zdata/Github/snippetbox$ go run ./cmd/web
time=2024-09-05T19:13:17.197+08:00 level=INFO msg="starting server" addr=:4000
time=2024-09-05T19:13:28.452+08:00 level=INFO msg="received request" ip=[::1]:57134 proto=HTTP/1.1 method=GET uri=/
2024/09/05 19:13:28 http: invalid Content-Length of "this isn't an integer!"

Unfortunately, it’s not possible to configure http.Server to use our new structured logger directly. Instead, we have to convert our structured logger handler into a *log.Logger which writes log entries at a specific fixed level, and then register that with the http.Server . We can do this conversion using the slog.NewLogLogger() function, like so:

func main() {
    addr := flag.String("addr", ":4000", "HTTP network address")
    dbDriver := flag.String("dbdriver", "mysql", "Database driver name")
    dsn := flag.String("dsn",
        "zeb:zebpwd@tcp(localhost:3306)/snippetbox?parseTime=true",
        "MySQL data source name")
    flag.Parse()

    logger := slog.New(slog.NewTextHandler(os.Stdout, nil))

    db, err := openDB(*dbDriver, *dsn)
    if err != nil {
        logger.Error(err.Error())
        os.Exit(1)
    }
    defer db.Close()

    templateCache, err := newTemplateCache()
    if err != nil {
        logger.Error(err.Error())
        os.Exit(1)
    }

    formDecoder := form.NewDecoder()

    sessionManager := scs.New()
    sessionManager.Store = mysqlstore.New(db)
    sessionManager.Lifetime = 12 * time.Hour

    app := &application{
        logger:         logger,
        snippet:        &models.SnippetModel{DB: db},
        templateCache:  templateCache,
        formDecoder:    formDecoder,
        sessionManager: sessionManager,
    }

    srv := &http.Server{
        Addr:    *addr,
        Handler: app.routes(),
        // Create a *log.Logger from our structured logger handler, which writes log entries at
        // Error level, and assign it to the ErrorLog field. If you would prefer to log the
        // server errors at Warn level instead, you could pass slog.LevelWarn as the final
        // parameter.
        ErrorLog: slog.NewLogLogger(logger.Handler(), slog.LevelError),
    }

    logger.Info("starting server", "addr", *addr)

    err = srv.ListenAndServe()
    logger.Error(err.Error())
    os.Exit(1)
}

 

zzh@ZZHPC:/zdata/Github/snippetbox$ go run ./cmd/web
time=2024-09-05T19:24:08.317+08:00 level=INFO msg="starting server" addr=:4000
time=2024-09-05T19:24:16.686+08:00 level=INFO msg="received request" ip=[::1]:47308 proto=HTTP/1.1 method=GET uri=/
time=2024-09-05T19:24:16.689+08:00 level=ERROR msg="http: invalid Content-Length of \"this isn't an integer!\""

 

Generating a self-signed TLS certificate

Let’s switch our attention to getting our application using HTTPS (rather than plain HTTP) for all requests and responses.

HTTPS is essentially HTTP sent across a TLS (Transport Layer Security) connection. The advantage to this is that HTTPS traffic is encrypted and signed, which helps ensure its privacy and integrity during transit.

Note: If you’re not familiar with the term, TLS is essentially the modern version of SSL (Secure Sockets Layer). SSL now has been officially deprecated due to security concerns, but the name still lives on in the public consciousness and is often used interoperably with TLS. For clarity and accuracy, we’ll stick with the term TLS.

Before our server can start using HTTPS, we need to generate a TLS certificate.

For production servers I recommend using Let’s Encrypt to create your TLS certificates, but for development purposes the simplest thing to do is to generate your own self-signed certificate.

A self-signed certificate is the same as a normal TLS certificate, except that it isn’t cryptographically signed by a trusted certificate authority. This means that your web browser will raise a warning the first time it’s used, but it will nonetheless encrypt HTTPS traffic correctly and is fine for development and testing purposes.

Handily, the crypto/tls package in Go’s standard library includes a generate_cert.go tool that we can use to easily create our own self-signed certificate.

Create a new tls directory in the root of your project repository to hold the certificate.

To run the generate_cert.go tool, you’ll need to know the place on your computer where the source code for the Go standard library is installed. If you’re using Linux, macOS or FreeBSD and followed the official install instructions, then the generate_cert.go file should be located under /usr/local/go/src/crypto/tls .

zzh@ZZHPC:/zdata/Github/snippetbox/tls$ go run ~/.goenv/versions/1.23.0/src/crypto/tls/generate_cert.go --rsa-bits=2048 --host=localhost
2024/09/05 19:38:42 wrote cert.pem
2024/09/05 19:38:42 wrote key.pem

Behind the scenes, this generate_cert.go command works in two stages:

1. First it generates a 2048-bit RSA key pair, which is a cryptographically secure public key and private key.

2. It then stores the private key in a key.pem file, and generates a self-signed TLS certificate for the host localhost containing the public key — which it stores in a cert.pem file. Both the private key and certificate are PEM encoded, which is the standard format used by most TLS implementations.

 

The mkcert tool

As an alternative to the generate_cert.go tool, you might want to consider using mkcert to generate the TLS certificates. Although this requires some extra set up, it has the advantage that the generated certificates are locally trusted — meaning that you can use them for testing and development without getting security warnings in your web browser.

 

Running a HTTPS server

Now that we have a self-signed TLS certificate and corresponding private key, starting a HTTPS web server is lovely and simple — we just need open the main.go file and swap the srv.ListenAndServe() method for srv.ListenAndServeTLS() instead.

func main() {
    addr := flag.String("addr", ":4000", "HTTP network address")
    dbDriver := flag.String("dbdriver", "mysql", "Database driver name")
    dsn := flag.String("dsn",
        "zeb:zebpwd@tcp(localhost:3306)/snippetbox?parseTime=true",
        "MySQL data source name")
    flag.Parse()

    logger := slog.New(slog.NewTextHandler(os.Stdout, nil))

    db, err := openDB(*dbDriver, *dsn)
    if err != nil {
        logger.Error(err.Error())
        os.Exit(1)
    }
    defer db.Close()

    templateCache, err := newTemplateCache()
    if err != nil {
        logger.Error(err.Error())
        os.Exit(1)
    }

    formDecoder := form.NewDecoder()

    sessionManager := scs.New()
    sessionManager.Store = mysqlstore.New(db)
    sessionManager.Lifetime = 12 * time.Hour
    // Make sure that the Secure attribute is set on our session cookies. Setting this means that 
    // the cookie will only be sent by a user's web browser when an HTTPS connection is being 
    // used (and won't be sent over an unsecure HTTP connection).
    sessionManager.Cookie.Secure = true

    app := &application{
        logger:         logger,
        snippet:        &models.SnippetModel{DB: db},
        templateCache:  templateCache,
        formDecoder:    formDecoder,
        sessionManager: sessionManager,
    }

    srv := &http.Server{
        Addr:    *addr,
        Handler: app.routes(),
        ErrorLog: slog.NewLogLogger(logger.Handler(), slog.LevelError),
    }

    logger.Info("starting server", "addr", *addr)

    // Use the ListenAndServeTLS() method to start the HTTPS server. We pass in the paths to the 
    // TLS certificate and corresponding private key as the two parameters.
    err = srv.ListenAndServeTLS("./tls/cert.pem", "./tls/key.pem")
    logger.Error(err.Error())
    os.Exit(1)
}

 

zzh@ZZHPC:/zdata/Github/snippetbox$ go run ./cmd/web
time=2024-09-05T19:56:14.409+08:00 level=INFO msg="starting server" addr=:4000
time=2024-09-05T19:56:27.312+08:00 level=ERROR msg="http: TLS handshake error from [::1]:60980: client sent an HTTP request to an HTTPS server"
time=2024-09-05T19:56:40.223+08:00 level=ERROR msg="http: TLS handshake error from [::1]:33054: remote error: tls: unknown certificate"
time=2024-09-05T19:56:40.230+08:00 level=ERROR msg="http: TLS handshake error from [::1]:33056: remote error: tls: unknown certificate"
time=2024-09-05T19:56:40.235+08:00 level=ERROR msg="http: TLS handshake error from [::1]:33072: remote error: tls: unknown certificate"
time=2024-09-05T19:56:40.238+08:00 level=INFO msg="received request" ip=[::1]:33084 proto=HTTP/2.0 method=GET uri=/
time=2024-09-05T19:56:40.253+08:00 level=INFO msg="received request" ip=[::1]:33084 proto=HTTP/2.0 method=GET uri=/static/css/main.css
time=2024-09-05T19:56:40.254+08:00 level=INFO msg="received request" ip=[::1]:33084 proto=HTTP/2.0 method=GET uri=/static/js/main.js
time=2024-09-05T19:56:40.279+08:00 level=INFO msg="received request" ip=[::1]:33084 proto=HTTP/2.0 method=GET uri=/static/img/logo.png
time=2024-09-05T19:57:06.569+08:00 level=INFO msg="received request" ip=[::1]:33084 proto=HTTP/2.0 method=GET uri=/snippet/create
time=2024-09-05T19:57:29.219+08:00 level=ERROR msg="http: TLS handshake error from [::1]:32772: EOF"
time=2024-09-05T19:57:29.219+08:00 level=ERROR msg="http: TLS handshake error from [::1]:60990: EOF"
time=2024-09-05T19:57:32.914+08:00 level=INFO msg="received request" ip=[::1]:33084 proto=HTTP/2.0 method=POST uri=/snippet/create
time=2024-09-05T19:57:32.925+08:00 level=INFO msg="received request" ip=[::1]:33084 proto=HTTP/2.0 method=GET uri=/snippet/view/9

 

HTTP requests

It’s important to note that our HTTPS server only supports HTTPS. If you try making a regular HTTP request to it, the server will send the user a 400 Bad Request status and the message "Client sent an HTTP request to an HTTPS server" . Nothing will be logged.

 

HTTP/2 connections

A big plus of using HTTPS is that Go’s will automatically upgrade the connection to use HTTP/2 if the client supports it.

This is good because it means that, ultimately, our pages will load faster for users.

 

Certificate permissions

It’s important to note that the user that you’re using to run your Go application must have read permissions for both the cert.pem and key.pem files, otherwise ListenAndServeTLS() will return a permission denied error.

By default, the generate_cert.go tool grants read permission to all users for the cert.pem file but read permission only to the owner of the key.pem file.

Generally, it’s a good idea to keep the permissions of your private keys as tight as possible and allow them to be read only by the owner or a specific group.

 

Source control

If you’re using a version control system (like Git or Mercurial) you may want to add an ignore rule so the contents of the tls directory are not accidentally committed. With Git, for instance:

zzh@ZZHPC:/zdata/Github/snippetbox$ echo 'tls/' >> .gitignore

 

Configuring HTTPS settings

Go has good default settings for its HTTPS server, but it’s possible to optimize and customize how the server behaves.

One change, which is almost always a good idea to make, is to restrict the elliptic curves that can potentially be used during the TLS handshake. Go supports a few elliptic curves, but as of Go 1.23 only tls.CurveP256 and tls.X25519 have assembly implementations. The others are very CPU intensive, so omitting them helps ensure that our server will remain performant under heavy loads.

To make this tweak, we can create a tls.Config struct containing our non-default TLS settings, and add it to our http.Server struct before we start the server.

import (
    "crypto/tls"
    ...
)

...

func main() {
    addr := flag.String("addr", ":4000", "HTTP network address")
    dbDriver := flag.String("dbdriver", "mysql", "Database driver name")
    dsn := flag.String("dsn",
        "zeb:zebpwd@tcp(localhost:3306)/snippetbox?parseTime=true",
        "MySQL data source name")
    flag.Parse()

    logger := slog.New(slog.NewTextHandler(os.Stdout, nil))

    db, err := openDB(*dbDriver, *dsn)
    if err != nil {
        logger.Error(err.Error())
        os.Exit(1)
    }
    defer db.Close()

    templateCache, err := newTemplateCache()
    if err != nil {
        logger.Error(err.Error())
        os.Exit(1)
    }

    formDecoder := form.NewDecoder()

    sessionManager := scs.New()
    sessionManager.Store = mysqlstore.New(db)
    sessionManager.Lifetime = 12 * time.Hour
    sessionManager.Cookie.Secure = true // Setting this means the cookie will only be sent by a 
                                        // user's web browser when an HTTPS connection is used.

    app := &application{
        logger:         logger,
        snippet:        &models.SnippetModel{DB: db},
        templateCache:  templateCache,
        formDecoder:    formDecoder,
        sessionManager: sessionManager,
    }

    // Initialize a tls.Config struct to hold the non-default TLS settings we want the server to 
    // use. In this case the only thing that we're changing is the curve preferences value, so 
    // that only elliptic curves with assembly implementations are used.
    tlsConfig := &tls.Config{
        CurvePreferences: []tls.CurveID{tls.X25519, tls.CurveP256},
    }

    // Set the server's TLSConfig field to use the tlsConfig variable we just created.
    srv := &http.Server{
        Addr:    *addr,
        Handler: app.routes(),
        ErrorLog: slog.NewLogLogger(logger.Handler(), slog.LevelError),
        TLSConfig: tlsConfig,
    }

    logger.Info("starting server", "addr", *addr)

    err = srv.ListenAndServeTLS("./tls/cert.pem", "./tls/key.pem")
    logger.Error(err.Error())
    os.Exit(1)
}

 

TLS versions

By default, Go’s HTTPS server is configured to support TLS 1.2 and 1.3. You can customize this and change the minimum and maximum TLS versions using the tls.Config.MinVersion and MaxVersion fields and TLS versions constants in the
crypto/tls package.

For example, if you want the server to support TLS versions 1.0 to 1.2 only, you can use a configuration like so:

tlsConfig := &tls.Config{
    MinVersion: tls.VersionTLS10,
    MaxVersion: tls.VersionTLS12,
}

 

Connection timeouts

Let’s take a moment to improve the resiliency of our server by adding some timeout settings, like so:

    srv := &http.Server{
        Addr:         *addr,
        Handler:      app.routes(),
        ErrorLog:     slog.NewLogLogger(logger.Handler(), slog.LevelError),
        TLSConfig:    tlsConfig,
        IdleTimeout:  time.Minute,
        ReadTimeout:  5 * time.Second,
        WriteTimeout: 10 * time.Second,
    }

All three of these timeouts — IdleTimeout , ReadTimeout and WriteTimeout — are server-wide settings which act on the underlying connection and apply to all requests irrespective of their handler or URL.

 

The IdleTimeout setting

By default, Go enables keep-alives on all accepted connections. This helps reduce latency (especially for HTTPS connections) because a client can reuse the same connection for multiple requests without having to repeat the TLS handshake.

By default, keep-alive connections will be automatically closed after a couple of minutes (the exact time depends on your operating system). This helps to clear-up connections where the user has disappeared unexpectedly — e.g. due to a power cut on the client’s end.

There is no way to increase this default (unless you roll your own net.Listener ), but you can reduce it via the IdleTimeout setting. In our case, we’ve set IdleTimeout to 1 minute, which means that all keep-alive connections will be automatically closed after 1 minute of inactivity.

 

The ReadTimeout setting

In our code we’ve also set the ReadTimeout setting to 5 seconds. This means that if the request headers or body are still being read 5 seconds after the request is first accepted, then Go will close the underlying connection. Because this is a ‘hard’ closure on the connection, the user won’t receive any HTTP(S) response.

Setting a short ReadTimeout period helps to mitigate the risk from slow-client attacks — such as Slowloris — which could otherwise keep a connection open indefinitely by sending partial, incomplete, HTTP(S) requests.

Important: If you set ReadTimeout but don’t set IdleTimeout , then IdleTimeout will default to using the same setting as ReadTimeout . For instance, if you set ReadTimeout to 3 seconds, then there is the side-effect that all keep-alive connections will also be closed after 3 seconds of inactivity. Generally, my recommendation is to avoid any ambiguity and always set an explicit IdleTimeout value for your server.

 

The WriteTimeout setting

The WriteTimeout setting will close the underlying connection if our server attempts to write to the connection after a given period (in our code, 10 seconds). But this behaves slightly differently depending on the protocol being used.

• For HTTP connections, if some data is written to the connection more than 10 seconds after the read of the request header finished, Go will close the underlying connection instead of writing the data.
• For HTTPS connections, if some data is written to the connection more than 10 seconds after the request is first accepted, Go will close the underlying connection instead of writing the data. This means that if you’re using HTTPS (like we are) it’s sensible to set WriteTimeout to a value greater than ReadTimeout .

It’s important to bear in mind that writes made by a handler are buffered and written to the connection as one when the handler returns. Therefore, the idea of WriteTimeout is generally not to prevent long-running handlers, but to prevent the data that the handler returns from taking too long to write.