Welcome to Learn Grafana 10.x! Together, we will explore Grafana, an exciting, multi-faceted visualization tool for data exploration, analysis, and alerting. We will learn how to install Grafana, become familiar with some of its many features, and even use it to investigate publicly available real-world datasets.

Whether you are an engineer watching terabytes of metrics for a critical system fault, an administrator sifting through a haystack of log output looking for the needle of an application error, or just a curious citizen eager to know how your city works, Grafana can help you monitor, explore, and analyze data. The key to getting a handle on big data is the ability to visualize it.

But before we find out how Grafana gives you that ability, we’ll need to cover a few basic concepts behind data visualization. Following that, we’ll set up our own instance of Grafana, which will form the fundamental building block for the exercises that follow in later chapters.

The following topics will be covered in this chapter:

• Appreciating data and visualization – we’ll take a brief overview of the data landscape and how visualization is useful
• Why Grafana? We’ll look at what makes Grafana an attractive solution
• Installing Grafana – we’ll install the Grafana application server and get it running
• Connecting to the Grafana server – we’ll launch the Grafana application by connecting to the installed server from a web browser

Tutorial code, dashboards, and other helpful files for this chapter can be found in the book’s GitHub repository at https://github.com/PacktPublishing/Learn-Grafana-10/tree/main/Chapter01.

Grafana is relatively easy to set up, but since it is a web server application, you will need to execute a few shell commands to get it running. For the purposes of this book, we will assume that you will access Grafana from the same computer that you installed it on. The following are the technical requirements for installing and running Grafana:

• Familiarity with the command shell
• A terminal application or an SSH to the machine where you plan to install Grafana
• Docker (in order to run Grafana from a Docker container)

Optionally, you will need to have the following:

• Administrator access to install and run Grafana from the command line, rather than in a Docker container

In the not-too-distant past, most of us consumed data pretty much solely via a daily newspaper—the financial pages, the sports section, and the weather forecast. However, in recent years, the ubiquity of computing power has immersed every part of our lives in a sea of data.

Around the clock, our built environment and devices collect innumerable amounts of data, which we consume. Our morning routine starts with a review of emails, social media posts, and news feeds on a smartphone or tablet, and whereas we once put down the daily newspaper when we left for work, our phones come with us everywhere.

We walk around or exercise and our phones capture our activity and location data via the global positioning system (GPS), while our smartwatches capture our vital signs. When we browse the web, every single interaction down to a mouse click is logged and stored for analysis. The servers that deliver these experiences are monitored and maintained by engineers on a round-the-clock basis. Marketers and salesforces continually analyze this data in order to make business-critical decisions.

On the way to work, our cars, buses, and trains contain increasingly sophisticated computers that silently log tens of thousands of real-time metrics, using them to calculate efficiency, profitability, engine performance, and environmental impact. Technicians evaluating these physical systems’ health or troubleshooting problems often sift through an enormous stream of data to tease out the signs of a faulty sensor or a failed part. The importance of this data is globally recognized. This is precisely why data recorders are the most valuable forensic artifact after any transportation accident, and why their recovery generates such widespread media coverage.

Meanwhile, in the modern home, a smart thermostat dutifully logs the settings on a Heating, Ventilation, and Air Conditioning (HVAC) system, as well as the current temperature both inside and outside the house. These devices continually gather real-time weather information in order to make decisions about how and when to run most efficiently.

Similar to the systems at home, but on a much larger scale, nearly every building we pass through during the day collects and monitors the health of a number of key infrastructure systems, from air conditioning to plumbing to security. No amount of paper could possibly record the thousands of channels of data flowing through these physical plants, and yet the building management system aggregates this data to make the same kinds of simple decisions as the homeowner does.

Moreover, these examples represent only a drop in the ocean of data. Around the world, governments, scientists, NGOs, and everyday citizens collect, store, and analyze their own datasets. They are all confronted with the same issue: how to aggregate, collate, or distill the mass of data into a form that a human can perceive and act on in a few seconds or less. The response to this issue is effective data storage and visualization.

Storing, retrieving, and visualizing data

For years, the basic language of data visualization was well-defined: using a chart, graph, histogram, and so on. What was missing was the ability to rapidly create these charts and graphs not in hours or days but in seconds or even milliseconds. This requires processing power that draws representations of thousands and thousands of data points in the time it takes to refresh a computer display.

For decades, only the most powerful computers could manage the processing power required to visualize data on this scale, and the software they ran was specialized and expensive. However, a number of trends in computing have converged to produce a renaissance in data acquisition and visualization, making it accessible not only to domain practitioners but also to technically proficient members of the general public. They are as follows:

• Cheap general-purpose CPUs and graphics GPUs
• Inexpensive high-capacity storage, optimized for physical size and maximum throughput
• Web standards and technologies, including JavaScript and CSS
• Open source software frameworks and toolkits
• Scalable cloud computation at affordable prices
• Broadband networking to enterprises, homes, and mobile devices

A common feature of virtually all of this data, that is, for each sample from a sensor or line in a log file, is the snapshot from an invisible ticking clock: a timestamp. A dataset gathered from these data points across a period of time is referred to as a time series. A stored object containing one or more time series is a time-series dataset. An application that can provide optimized access to one or more of these datasets is called, naturally, a time-series database (TSDB). While a whole class of NoSQL time-series databases, such as InfluxDB, OpenTSDB, and Prometheus, have sprung up, venerable SQL relational databases, such as PostgreSQL and MySQL, have added their own support for time-series datasets.

That’s fine for storing and retrieving data, but what about visualizing data? Enter Grafana.

While there are many solutions in the data visualization space, Grafana is proving to be one of the most exciting, exhibiting rapid growth in scope and features, broad options for deployment and support, and an enthusiastic community contributing to its future growth. Before going into the specific features that make Grafana an attractive solution, let’s take a look at the criteria we might use to characterize a useful data visualization application:

Figure 1.1 – Grafana UI

For the purposes of this book, we will be looking at particular software applications that fulfill four major functions: exploration, analysis, presentation, and observability.

Exploration

Quickly loading and displaying a dataset with the idea of identifying the particularly interesting features for deeper analysis, sometimes referred to as drilling down, is an example of data exploration.

Another common term for data exploration is ad hoc analysis. This refers to the nature of using data visualization techniques without a pre-defined analysis in place. Ad hoc analysis is useful for getting a feel for the data’s characteristics, and whether any interesting patterns are discernable.

Figure 1.2 – Grafana exploration

In this book, we’ll be frequently using the Explore feature of Grafana to perform just this sort of data exploration.

Analysis

After we have examined our data, we may well want to analyze it. That is, we may want to quantify the data statistically or correlate it with other data. For example, we may want to see what the maximum value or average value is, or otherwise aggregate the data for a specific time range. We may also want to look at multiple datasets over the same time period to look for events that might be time-correlated.

Figure 1.3 – Grafana analysis

Grafana contains several analysis features that we highlight throughout the book. We will also leverage Grafana’s powerful transformation features to aid us in our analysis.

Presentation

Once we have identified the data we are interested in, we will want to present it in an aesthetically pleasing manner that also gives the viewer clarity about what the data represents, in effect helping to tell a story about the data, which would be otherwise difficult to do without specific domain knowledge.

Figure 1.4 – Grafana presentation

Assembling panels into dashboards is a common Grafana workflow for presentation, and we will spend much of our time in this book covering not only how to construct dashboards to tell the story of our data, but also how to structure our data visualizations to be both clear and meaningful.

Observability

Finally, we may need to observe the data over time, or even in real time as it may represent critical data. If the data crosses into a realm of concern, we may need to be notified immediately.

Grafana has extensive and powerful observability features, along with integrations for popular notification services such as PagerDuty. In this book, we’ll learn how to build alerts to detect anomalies in our data, and how to craft appropriate notifications depending upon the severity of the alert.

Choosing Grafana

While there are quite a few powerful data analytics tools on the market that fulfill these functions, Grafana has a number of features that make it an attractive choice:

• Fast: The Grafana backend is written in Google’s exciting Go language, making it extremely performant when querying data sources or feeding thousands of data points to multiple dashboard panels.
• Open: Grafana supports a plugin model for its dashboard panels and data sources. The number of plugins is constantly growing as the Grafana community enthusiastically contributes to the project.
• Beautiful: Grafana leverages the attractive and powerful D3 library. Many of the popular dashboard tools, such as Datadog and Zabbix, can quickly generate beautiful graphs from thousands of data channels, but they only offer some limited control over the display elements. Grafana provides fine-grained control over most graph elements including axes, lines, points, fills, annotations, and legends. It even offers the much sought-after dark mode.
• Versatile: Grafana is not tied to a particular database technology. For example, Kibana is a powerful, well-known member of Elasticsearch’s Elasticsearch, Logstash, and Kibana (ELK) stack; it is only capable of visualizing Elasticsearch data sources. This gives it the advantage over Grafana of a better ability to integrate Elasticsearch’s analysis tools in its graphing panels. However, due to its plugin architecture, Grafana can support a variety of ever-growing data sources (at last count in 2022, over 150), from traditional RDBMs, such as MySQL and PostgresQL, to modern TSDBs, such as InfluxDB and Prometheus. Not only can each graph display data from a variety of data sources, but a single graph can also combine data from multiple data sources.
• Free: While they are very powerful tools indeed, Datadog and Splunk are commercial packages and, as such, charge fees to manage all but the smallest datasets. If you want to get your feet wet, Grafana is freely available under the Apache open source license, and if you do plan to run it in your enterprise, you can purchase tiered support.

These are just some of the criteria you might use to evaluate Grafana against similar products. Your mileage may vary, but now is a great time to be in the market for visualization tools. Grafana and its competitors each have their own strengths and weaknesses, but they are all very capable applications. Here’s a short list of the few we covered:

• Kibana (https://www.elastic.co/)
• Splunk (www.splunk.com)
• Datadog (https://www.datadoghq.com/)
• Zabbix (https://www.zabbix.com/)

With this in mind, let’s install Grafana.

At its core, Grafana runs as a web server, and as such, it is not a typical double-click application. You will need to be comfortable with the command line and have administrator privileges on the computer you plan to install Grafana on. To download the latest versions of Grafana, check out https://grafana.com/grafana/download.

The Grafana application server runs on *nix operating systems (Linux, OS X, and Windows), and it can be installed locally on a laptop or workstation or on a remote server. It is even available as a hosted application if you’d rather not deal with setting up or managing a server application on your own.

In this section, we’ll walk through the most typical installation options:

• Docker
• OS X
• Linux
• Windows
• Hosted Grafana in the cloud

Once you’ve completed the installation of your choice, proceed to the Connecting to the Grafana server section for instructions on how to access Grafana from a web browser.

Grafana in a Docker container

The easiest and least complex installation method is to run Grafana from within a Docker container. Docker is available for all major platforms and can be downloaded by visiting https://www.docker.com/.

After installing Docker, open a terminal window and type in the following command:

% docker run -d --name=grafana -p 3000:3000 grafana/grafana

The percent (%) symbol is simply there to indicate we are typing in commands to an interactive shell such as zsh or Windows PowerShell. If you are cutting and pasting from the book, you’ll want to leave out that symbol.

Docker will automatically download and run the latest version of Grafana for your computer’s architecture. Bear in mind that since this basic container has no persistent storage, nothing will be retained if you delete the container. I suggest you run the container with a temporary volume so that Grafana’s internal database will continue to exist, even if you destroy the container:

% docker volume create grafana-storage
% docker run -d --name=grafana -p 3000:3000  \
    -v grafana-storage:/var/lib/grafana \
    grafana/grafana

Note

The book and its tutorial examples were written for the macOS operating system, a POSIX-compliant OS that shares many similarities to Linux, including the shell. However, with a few syntax modifications here and there, Windows users should be able to use these same commands in PowerShell.

For example, in the preceding command, you’ll want to use the backtick (`) in PowerShell for line continuation, rather than the backslash (\).

I will proceed with Docker and its companion product Docker Compose for the purposes of this book as it will allow an almost turnkey installation experience, as all the necessary dependencies will be automatically downloaded with the container. It will also install in its own sandbox, so you don’t need to worry about installing a stack of software that will be difficult to delete later. Finally, in future chapters, we will be setting up data sources using similar Docker containers, so managing the data pipeline as a combination of containers will be very consistent and straightforward.

Make and Makefile

In the book’s GitHub repository, you’ll find a Makefile in each chapter directory. You can use it to streamline some of the common Docker commands. If you’re not familiar with the make command or it isn’t installed on your computer, you can still cut and paste many of the commands embedded in the Makefile.

While space doesn’t permit a comprehensive introduction to the concepts that underlie make, here is a quick example of how to use it in concert with this book. The following is from the Makefile in the Chapter02 directory of the book’s GitHub repository:

up:
    docker-compose up -d --pull missing

On the first line, the word up before the colon (:) is referred to as the target. Anything following that colon is a dependency; there are no dependencies associated with the target. The second line is the command; there must be at least one command. For decades, the venerable make command and associated Makefile have been the backbone for building software, often with hundreds of complex dependencies. Nonetheless, for our use of make, we’ll use it mostly as a notepad of shortcut commands. To use it, you simply run a make command from the shell:

% make <target>

make will first run any targets named in the list of dependencies, followed by the command(s) associated with the target. Run the following:

% make up

You are using make to run this equivalent command:

% docker-compose up -d --pull missing

There is no requirement to use make for this book; all the commands you need are in the text.

Grafana for macOS

There are two options for installing and running Grafana for macOS:

• Homebrew
• Command line binary install

Using Homebrew is the simplest option as it wraps all the installation chores in a single command. To get Homebrew, visit https://brew.sh/. If you want more control over where to install Grafana, the command line option is a better choice.

Homebrew

Homebrew does not ship as part of macOS, but you can easily install it:

% /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

To install Grafana via Homebrew, run the following commands:

% brew update
% brew install grafana

If you want to keep Grafana running even after a reboot, use the Homebrew services subcommand to launch the installed Grafana application as a service. You will first need to confirm services installation:

% brew tap homebrew/services
% brew services start grafana

Command line binary install

To install via the command line, open a Terminal shell window and download the macOS distribution tarball, then untar it into the directory of your choice (replace ${GRAFANA_VERSION} with the current version):

% wget https://dl.grafana.com/oss/release/grafana-${GRAFANA_VERSION}.darwin-amd64.tar.gz
% tar -zxvf grafana-$GRAFANA_VERSION.darwin-amd64.tar.gz

Once you’ve untarred the file, cd into the directory and launch the binary by executing this command:

% ./bin/grafana-server web

Grafana for Linux

While Linux comes in a number of flavors, each falls into one of two installation systems: yum for Red Hat-based releases or apt for Debian or Ubuntu releases. Typically, you download the binary and then run the installer on the package file. To get the latest Grafana binaries for Linux, visit https://grafana.com/grafana/download?platform=linux.

Yum installation (Red Hat, Fedora, CentOS)

The installer for Red Hat distributions (CentOS, Fedora, and Red Hat) is yum. To download and install it (replace ${GRAFANA_VERSION} with the current version), use the following:

% wget https://dl.grafana.com/oss/release/grafana-%{GRAFANA_VERSION}.x86_64.rpm
% sudo yum install grafana-${GRAFANA_VERSION}.x86_64.rpm

To start up Grafana, use systemctl:

% systemctl daemon-reload
% systemctl start grafana-server
% systemctl status grafana-server

To keep Grafana running even after a reboot, use the following:

% sudo systemctl enable grafana-server.service

Apt installation (Debian, Ubuntu)

The installer for the Debian distributions (Debian and Ubuntu) is dpkg. To download and install it (replace ${GRAFANA_VERSION} with the current version), use the following:

% sudo apt-get install -y adduser libfontconfig1
% wget https://dl.grafana.com/oss/release/grafana_${GRAFANA_VERSION}_amd64.deb
% sudo dpkg -i grafana_${GRAFANA_VERSION}_amd64.deb

To start up Grafana, use the following:

% systemctl daemon-reload
% systemctl start grafana-server
% systemctl status grafana-server

To keep Grafana running even after a reboot, use the following:

% sudo systemctl enable grafana-server.service

Grafana for Windows

Installation for Windows is straightforward:

1. Go to https://grafana.com/grafana/download?platform=windows.
2. Download the latest MSI installer file from the download link.
3. Launch the .msi file to install.

Grafana Cloud

If you would rather not install Grafana on your computer, or you don’t have access to a computer that can run Grafana, there is another option—Grafana will host a free instance for you. Free Grafana Cloud hosting provides very generous limits on the number of users, metrics, logs, and traces. To sign up for the hosted version, go to https://grafana.com/get/ and select Cloud.

Now that you have installed and started up Grafana, let’s have a look at the interface. Grafana is a web application, so we’ll connect to it with an ordinary web browser such as Chrome, Safari, or Edge.

Once you have installed and launched Grafana, open a browser page to access the Grafana application. It can be found at http://localhost:3000. If everything goes well, you should see a login page, as follows:

Figure 1.5 – Grafana login

Log in with the admin username with the password admin. You will then be prompted to change it to something more secure (which you can skip if you wish). Once you have logged in, you should see the base Grafana interface:

Figure 1.6 – Grafana home page

Great job! You’ve successfully installed and connected the Grafana application.

Congratulations! Over the course of this chapter, we learned about data visualization and why Grafana is a powerful tool for data visualization. We also downloaded and installed Grafana. Finally, we launched the Grafana application from our browser, setting us on a learning path for future chapters.

In our next chapter, we’ll take a tour of the Grafana interface and familiarize ourselves with its basic features. This will serve as a foundation for upcoming tutorial exercises. I’m looking forward to our shared journey!

The official Grafana documentation can be found on their website at https://grafana.com/docs/.