Difference between revisions of "HABMC"

From Stanford SSI Wiki
Jump to navigation Jump to search
 
(3 intermediate revisions by the same user not shown)
Line 1: Line 1:
HABMC is the online mission control from which we can monitor balloons launches. It updates automatically in real time.
+
The Stanford Student Space Initiative has launched dozens of high altitude balloons. These balloons can go up to 100,000 feet or travel as far as Morocco. Yet these flights would hardly be valuable without the ability to read, parse, and visualize the data streaming back to the ground.
 +
 
 +
The data comes from sensors on board the balloon, which is then heavily compressed and sent over the Iridium satellite network. Naturally, what exactly that data is varies with each flight, which means that HABMC has to be extremely flexible. In addition to visualizing the telemetry data sent on every flight, HABMC has to be capable of everything from displaying ozone concentrations to detecting voltage spikes -- all in real time.
 +
 
 +
== How to get involved ==
 +
At a balloons meeting, talk to Kai, or just talk to him on slack {{slack-user|kai|}}.
 +
You can also check out our tutorials [https://github.com/stanford-ssi/habmc-tutorials]
  
 
== Public Page ==
 
== Public Page ==
 
The public page of HABMC shows the position and altitude of the balloon, as provided by the ValBal communications system. This allows us to provide detailed information to the FAA and other viewers without overwhelming them with a vast quantity of potentially misleading data.
 
The public page of HABMC shows the position and altitude of the balloon, as provided by the ValBal communications system. This allows us to provide detailed information to the FAA and other viewers without overwhelming them with a vast quantity of potentially misleading data.
[[File: HABMC_Public.png | 500px | right]]
 
  
 
== SSI Interface ==
 
== SSI Interface ==
If you're logged in as a member of the space initiative or have an access code, you can view the full SSI page. This means that we have better control over our historical data and a variety of utilities.
+
If you're logged in as a member of the space initiative or have an access code, you can view the full SSI page, jam-packed with features.
  
 
=== Dashboard ===
 
=== Dashboard ===
This page has an overview of the flight so far. The first row has the current altitude, ascent rate, temperature, ballast drops, vents, and battery voltage, along with each of their changes since the last transmission and a graph of the last several data points. The next row contains a map of the most recent transmissions, and a map of the whole flight thus far. The third row has indicators for remaining ballast, cutdown state, and battery state. The fourth row consists of a graph displaying a couple of the most important metrics together.
+
The dashboard provides a quick overview of the flight. Fully customizable for each payload, it displays a series of graphs to understand recent trends, underneath which is a map of the flight and charts detailing remaining quantities such as battery and ballast. Below the map lies a table of the configuration on the payload itself that gives exact numbers about how the payload is set to behave.
 +
 
 +
[[File: Dashboard_world_record.png | 500px | center]]
 +
 
 +
=== Maps, Charts, & Graphs ===
 +
We often want finer control over how we delve into and analyze the data. To that end, we built the charts utility, capable of graphing an arbitrary number of variables against each other. The user may zoom into the data by time period, or look at the whole mission.
 +
 
 +
[[File: Charts.png | 500px | center]]
 +
 
 +
Naturally, to see position data we have maps as well. On most flights we have three redundant positioning systems: a primary GPS baked into the board, an external GPS, and position estimates given by the Iridium satellite network. Each of these systems has their own foibles; for example, HABMC is capable of filtering out erroneous Iridium position estimates and displaying error bounds on the map itself.
 +
 
 +
=== Raw Data Viewing ===
 +
Sometimes charts, maps, and graphs aren’t enough and the flight controllers need to read the data directly. To that end, HABMC provides tables of all transmissions, the raw, compressed details, and every other piece of data the suite has access to. To make some of this data more readable, it shows the most recent transmission’s data grouped and sorted appropriately; one can also browse through old transmissions in this mode. Finally, should the flight controllers want to access the data directly, HABMC offers a full API.
 +
 
 +
=== Messages ===
 +
HABMC can also send messages back to the balloon, in addition to getting data from it. These messages, however, are far from human readable. HABMC builds these messages through a more intuitive interface, then verifies that they’re in the correct format before ultimately sending them to our payload. It will also store these messages and notify the flight controllers so as to ensure that everyone stays informed.
 +
 
 +
=== Configuration and permissions ===
 +
From the start, HABMC was designed to handle a wide variety of flight profiles. It’s able to understand three different parsing formats; much of the UI is even customizable. HABMC can even be set to notify the flight controllers via Slack or SMS when it detects a suspicious transmission. To safeguard these features, HABMC has a sophisticated user management system to grant select permissions only to those who need them.
 +
 
 +
=== Prediction & Footprint Calculator ===
 +
We almost always want to know where the balloon will go. To that end, we have a full predictor utility. Although it currently relies on the GFS weather model, it’s designed to be compatible with the more-accurate ECMWF model should we gain access. The predictor, written from the group up in Rust, takes this data and turns it into real predictions.  
  
=== Full Map ===
+
[[File: Predictor.png | 500px | center]]
This page provides a full look at the flight so far, much larger than the version on the dashboard.
 
  
=== Transmissions ===
+
Of course, there are a lot of uncertainties when it comes to prediction, such as descent rate and burst altitude. Thus, when we want to know where the payload will land, we use HABMC’s footprint calculator. This system generates a probability distribution about where the balloon will ultimately come down.
This is a table all parsed transmissions that have come in so far.
 
  
=== Send Message ===
+
[[File: Footprint.png | 500px | center]]
This interface allows you to send messages to the balloon. In addition to having a place to enter the message, it shows the encoded version of the message and the cost to send. It requires both a set of rockblock credentials and to be logged in as an administrator. It then confirms with you before actually sending the message to avoid incorrect calls.  
 
  
=== Sent Messages ===
+
=== Navigation ===
This is just a table of all messages that have been sent to the balloon on this mission. Currently only administrators can view it.
+
One of the core features of our novel altitude stabilization platform, ValBal, is that it’s capable of flying at whatever altitude we tell it to, and adjusting that altitude mid-flight. Since there are different wind patterns at different altitudes, by strategically adjusting altitudes the balloon can optimize ground distance or even aim for a specific location. For example, the balloon might fly at 13km to travel west, then ascend to 15km briefly to steer north toward its final destination.
  
=== SPOT Pages ===
+
[[File: Navigation.png | 500px | center]]
These pages each have an embedded map showing where the spot GPS is. There are four, one for each SPOT.
 
  
=== Iridium Transmissions ===
+
In this screenshot, the naive option of flying at the single altitude with the fastest winds is shown in blue; active guidance is shown in green. In the same total flight time, with active guidance the balloon can fly over twice as far.  
This page provides a look at the raw, encoded version of the messages we have received from the balloon, in addition to some metadata that RockBlock provides us.
 
  
=== Iriridium Map ===
+
=== Commands Module ===
This map shows us the position of the balloon, based on where the Iridium Satellite Network thinks it is. It also shows a circle giving the estimated accuracy of the position. Given that this has very low acccuracy, typically close to 12km, this should only be used if all other positioning systems go down.
+
On the launch site, we often suffer from an unreliable internet connection -- yet we still need to know how our payload is performing. To that end, HABMC also hosts a natural language processing module. While we may not have internet, we almost always have SMS service: the commands module allows us to text commands and queries to HABMC. Example messages include “When was the last transmission?” or “Push the launch time back by 15 minutes”, to which HABMC will respond appropriately.  
  
=== Predictor ===
+
=== Livestreaming ===
This allows the user to run predictions on the balloon, from its current position, using manually input parameters of altitude and duration. If these parameters are not provided, it takes defaults from the last recorded position of the balloon.  
+
Sending live video down through SSTV can be displayed on HABMC as well. Piping the data through ffmpeg on the client to Youtube Live lets us maintain maximum compatibility with inputs and deliver the video at maximum spee
It then plots the prediction on the map, while keeping all prior predictions as well. One is able to remove these predictions with a clear button.
 
  
=== Charts and Graphs ===
+
[[File: Livestream.png | 500px | center]]
This page allows the plotting of any values versus any other given values. For example, the user can pick an “X axis” value, where logical options might be time or distance from launch site, and can then plot any number of other values on the Y axis. This could include voltage, solar current, velocity, altitude, temperature, etc (nearly any valuable flight parameter).
 
  
=== Configuration ===
+
=== High-bandwidth radio backend ===
This is a page that takes a bunch of parameters and settings for operation of the mission. In general, it is just a form that allows admins to set things like the format of the message, the flight ceiling and floor, and the maximum ballast capacity that are used to understand what the transmissions really mean.  
+
Capable of storing data and delivering high-bandwidth transmissions as fast as the database can handle, the radio code turns HABMC into a truly real time control suite. With the system in place, it can display the data from multiple transmissions per second.  
  
== Technical Background ==
+
[[File:Live-graphs.gif | 500px | center]]
HABMC is written with Ruby on Rails and Angular.js. While in a previous iteration the backend was written with PHP and hosted on GoDaddy, it is currently hosted on Heroku. We have a Postgres database and use NewRelic for analytics.
 
  
So that the data can updated with minimal latency, we open up a websocket connection from the client to the server. Then, as soon as the data changes on the server, such as when a new transmission comes in, a message is sent through the websocket back to the client that tells it to refresh its data. This refresh is done with a HTTP request, rather than sending the data through the websocket, because HTTP requests are much more developed than websockets. These HTTP requests are able to support functionality like authorization, CORS, and caching easily, in addition to avoiding repeated code on the initial load of the page.
+
== Technical Facts ==
 +
HABMC is written with Ruby on Rails and React.js, hosted on Heroku. External services include a prediction and navigation server (written in Rust), an InfluxDB for storing high-bandwidth radio transmissions, a Postgres primary database, writer and followers for radio transmission (written in Node), Newrelic for analytics, and Cloudflare as a CDN.
  
 
== History ==
 
== History ==
Line 66: Line 85:
 
A host of new features arrived between Version 1 and Version 2, like a login system and plotting FAA zones on maps. It also included an architectural change to reduce server load: rather than pinging the server constantly, we moved to a model with push notifications, based on websockets. To further reduce the load, we also batched requests when a change was triggered.
 
A host of new features arrived between Version 1 and Version 2, like a login system and plotting FAA zones on maps. It also included an architectural change to reduce server load: rather than pinging the server constantly, we moved to a model with push notifications, based on websockets. To further reduce the load, we also batched requests when a change was triggered.
  
== Version 3.0 ==
+
=== Version 3.0 ===
 
The frontend underwent a complete overhaul, getting rewritten from the ground up in React to be as efficient as possible. This made all of HABMC snappier -- and also way more maintainable. While we were at it, we redid the styles, switching to a dark theme.  
 
The frontend underwent a complete overhaul, getting rewritten from the ground up in React to be as efficient as possible. This made all of HABMC snappier -- and also way more maintainable. While we were at it, we redid the styles, switching to a dark theme.  
  

Latest revision as of 07:33, 19 September 2017

The Stanford Student Space Initiative has launched dozens of high altitude balloons. These balloons can go up to 100,000 feet or travel as far as Morocco. Yet these flights would hardly be valuable without the ability to read, parse, and visualize the data streaming back to the ground.

The data comes from sensors on board the balloon, which is then heavily compressed and sent over the Iridium satellite network. Naturally, what exactly that data is varies with each flight, which means that HABMC has to be extremely flexible. In addition to visualizing the telemetry data sent on every flight, HABMC has to be capable of everything from displaying ozone concentrations to detecting voltage spikes -- all in real time.

How to get involved

At a balloons meeting, talk to Kai, or just talk to him on slack SlackLogo.png@{{{display-name}}} . You can also check out our tutorials [1]

Public Page

The public page of HABMC shows the position and altitude of the balloon, as provided by the ValBal communications system. This allows us to provide detailed information to the FAA and other viewers without overwhelming them with a vast quantity of potentially misleading data.

SSI Interface

If you're logged in as a member of the space initiative or have an access code, you can view the full SSI page, jam-packed with features.

Dashboard

The dashboard provides a quick overview of the flight. Fully customizable for each payload, it displays a series of graphs to understand recent trends, underneath which is a map of the flight and charts detailing remaining quantities such as battery and ballast. Below the map lies a table of the configuration on the payload itself that gives exact numbers about how the payload is set to behave.

Dashboard world record.png

Maps, Charts, & Graphs

We often want finer control over how we delve into and analyze the data. To that end, we built the charts utility, capable of graphing an arbitrary number of variables against each other. The user may zoom into the data by time period, or look at the whole mission.

Charts.png

Naturally, to see position data we have maps as well. On most flights we have three redundant positioning systems: a primary GPS baked into the board, an external GPS, and position estimates given by the Iridium satellite network. Each of these systems has their own foibles; for example, HABMC is capable of filtering out erroneous Iridium position estimates and displaying error bounds on the map itself.

Raw Data Viewing

Sometimes charts, maps, and graphs aren’t enough and the flight controllers need to read the data directly. To that end, HABMC provides tables of all transmissions, the raw, compressed details, and every other piece of data the suite has access to. To make some of this data more readable, it shows the most recent transmission’s data grouped and sorted appropriately; one can also browse through old transmissions in this mode. Finally, should the flight controllers want to access the data directly, HABMC offers a full API.

Messages

HABMC can also send messages back to the balloon, in addition to getting data from it. These messages, however, are far from human readable. HABMC builds these messages through a more intuitive interface, then verifies that they’re in the correct format before ultimately sending them to our payload. It will also store these messages and notify the flight controllers so as to ensure that everyone stays informed.

Configuration and permissions

From the start, HABMC was designed to handle a wide variety of flight profiles. It’s able to understand three different parsing formats; much of the UI is even customizable. HABMC can even be set to notify the flight controllers via Slack or SMS when it detects a suspicious transmission. To safeguard these features, HABMC has a sophisticated user management system to grant select permissions only to those who need them.

Prediction & Footprint Calculator

We almost always want to know where the balloon will go. To that end, we have a full predictor utility. Although it currently relies on the GFS weather model, it’s designed to be compatible with the more-accurate ECMWF model should we gain access. The predictor, written from the group up in Rust, takes this data and turns it into real predictions.

Predictor.png

Of course, there are a lot of uncertainties when it comes to prediction, such as descent rate and burst altitude. Thus, when we want to know where the payload will land, we use HABMC’s footprint calculator. This system generates a probability distribution about where the balloon will ultimately come down.

Footprint.png

Navigation

One of the core features of our novel altitude stabilization platform, ValBal, is that it’s capable of flying at whatever altitude we tell it to, and adjusting that altitude mid-flight. Since there are different wind patterns at different altitudes, by strategically adjusting altitudes the balloon can optimize ground distance or even aim for a specific location. For example, the balloon might fly at 13km to travel west, then ascend to 15km briefly to steer north toward its final destination.

Navigation.png

In this screenshot, the naive option of flying at the single altitude with the fastest winds is shown in blue; active guidance is shown in green. In the same total flight time, with active guidance the balloon can fly over twice as far.

Commands Module

On the launch site, we often suffer from an unreliable internet connection -- yet we still need to know how our payload is performing. To that end, HABMC also hosts a natural language processing module. While we may not have internet, we almost always have SMS service: the commands module allows us to text commands and queries to HABMC. Example messages include “When was the last transmission?” or “Push the launch time back by 15 minutes”, to which HABMC will respond appropriately.

Livestreaming

Sending live video down through SSTV can be displayed on HABMC as well. Piping the data through ffmpeg on the client to Youtube Live lets us maintain maximum compatibility with inputs and deliver the video at maximum spee

Livestream.png

High-bandwidth radio backend

Capable of storing data and delivering high-bandwidth transmissions as fast as the database can handle, the radio code turns HABMC into a truly real time control suite. With the system in place, it can display the data from multiple transmissions per second.

Live-graphs.gif

Technical Facts

HABMC is written with Ruby on Rails and React.js, hosted on Heroku. External services include a prediction and navigation server (written in Rust), an InfluxDB for storing high-bandwidth radio transmissions, a Postgres primary database, writer and followers for radio transmission (written in Node), Newrelic for analytics, and Cloudflare as a CDN.

History

Origin

Kirill Safin first created HABMC, designed as a way of understanding ValBal launches. At this stage, functionality was minimal; it had the skeleton of the entire site, written with angular. The only pages that had any content were the SPOT GPS pages with their embedded maps displaying where the balloons were. Beyond that, the core achievement was the skeleton itself: it was well designed and provided an excellent foundation for what was to come.

Version 1.0

The next stage of HABMC, when it was still written with PHP, involved fleshing out the skeleton. We built the dashboard, which contains graphs and indicators of the flight vitals. The full map, which provided a larger view of the flight path, the transmissions page, which listed the parsed transmissions, and a utility to send messages to the balloon through RockBlock were all added as well. We created a couple pages to see what was happening at a lower level: a page to show us the raw, unparsed transmissions and a page that mapped the position estimate based on the iridium satellite network. One of the more exciting things that we added was the predictor utility, which made an educated guess as to where the balloon would go next, based off NOAA data. And to delve deeper into the data, we created the charts and graphs page, which graphed arbitrary data streams against each other. Finally, to control it all, we had a configuration page.

To power all of this, we had a complex backend, written at this point in PHP. When a transmission came in, it would be stored in a SQL database. Then, to parse and understand it, the transmission would be compared with the parser configuration we had set from the frontend. The backend was also responsible for keeping track of that configuration and making calculations (such as velocity) that were not in the transmission.

The data would also update in realtime, but it did so much less efficiently than it does now. Pages would repeatedly ping the backend and update their data if necessary. While this worked well from the user's perspective, as they never had to reload the page to see new data, it was terrible from the server's perspective. Each chart or graph, of which there were nine on the dashboard alone, had to make a request to the server every few seconds, meaning that the backend was bombarded with requests, most of which did nothing.

In this stage too a Github was created for easier code maintenance. While the deploy process still involved manually uploading the files via FTP, this simplified the challenges of working with multiple people on one project.

Version 2.0

The release of version 2 involved turning HABMC from a clunky interface to a more elegant bundle of code. To start, we rewrote the entire backend in Ruby on Rails. As GoDaddy does not support Rails apps, we transitioned to hosting HABMC on Heroku. Advantages here included better load monitoring and simpler deploys. The real advantages, though, came from the updated backend: we had a much more MVC oriented architecture, maintenance was easier, and security was stronger. Further, this allowed us to refactor the JavaScript that powered the core of HABMC. Prior to this, all of the JavaScript lived in one behemoth of a file. Taking advantage of the asset pipeline meant that we could separate this out into a couple dozen distinct files, making the codebase as a whole much simpler and understandable. Routing these files through the asset pipeline additionally meant that we could first minify and then gzip them (and all other static assets such as stylesheets), meaning that the end files were about a tenth the size.

A host of new features arrived between Version 1 and Version 2, like a login system and plotting FAA zones on maps. It also included an architectural change to reduce server load: rather than pinging the server constantly, we moved to a model with push notifications, based on websockets. To further reduce the load, we also batched requests when a change was triggered.

Version 3.0

The frontend underwent a complete overhaul, getting rewritten from the ground up in React to be as efficient as possible. This made all of HABMC snappier -- and also way more maintainable. While we were at it, we redid the styles, switching to a dark theme.

Future Plans

At this point, the core functionality of HABMC has stabilized, but there's always more to be done. Throughout the pipeline, HABMC can improve; old features need to be made more efficient, and new features need to be built. Many features, like navigation and radio integrations, need to mature to support the rigour of flight.

Above all, going into the future HABMC will continue the relentless cycle of refinement and improvement to build it into an ever more powerful mission control suite.

Balloon Launches
2014-15 SSI-19202122
2015-16 SSI-23(a) • 2425262728293031323334353637383940414243
2016-17 444546474849505152
2017-18
2018-19 8386879091
2019-20 929397
VE