...or, what happens when an app company runs buses...
Smartbus
Citymapper is an app company building a popular transport app for cities. For years we've helped city dwellers to get around by showing them the best public transport options available. In May 2017 we decided to run buses ourselves to really help users to get from A to B.
We find gaps in transit networks by analysing anonymised user data. Then we fill that gap by launching our own bus service.
In addition to filling a gap we also aimed to create a great user experience for buses to match our app. We painted them green and added our own unique branding and music
One of the main user facing features was a big internal screen informing passengers about the remaining time to upcoming stops and other relevant transit information. We also aimed to create an "app meets bus" experience to show off our cool tech which interacts with our systems in real time. The screen also told you when to get off by assigning you an emoji which you could track in the app.
Hardware
First of all we had to figure out which hardware to use for the buses. After a quick investigation it became clear that standard bus software is not up for this job - too inflexible and closed down. This hardware is custom built and is a great solution for big, non-technical bus operators but wouldn't have allowed us to achieve our set goal. Instead we bought consumer-grade commodity hardware that allowed us to make the setup cheaper. 4G router Teltonika Rut955 for internet connectivity. It also came in a great ruggedised case with lots of antennas - making our setup look super serious Android tablet running our own tracking and driver information app Raspberry Pi to power the screen. They are great little machines and very cheap A Raspberry Pi case that makes it fit onto a standardised DIN rail - to securely install our hardware in the vehicles The internal screen - together with a ruggedised case Some other cool tech that we experimented with like passenger counter infrared sensors, public wifi modules as well as the external facing headsigns (translation for non-bus geeks: headsigns are the displays at the front and side of buses showing where they are going). * A bunch of network cables and power adapters (big buses run on 24V, small ones on 12V, tablets and raspberry Pis on 5V...)
The tablet provided a GPS location to the Pi in order to make the screen show relevant information. Being able to do local networking on the bus allowed us to do this even when the internet connection failed.
Software
As soon as the hardware arrived we setup a 'bus lab' in the corner of our office - a full replica of the tech on the bus, complete with driver seat & tablet, a screen, power converters and lots of cables. Having this fake bus setup in our office was great as it allowed us to test our tech before going out to our buses and install anything there. If we cannot make it work in the office - how the hell is it going to work on a moving bus with flaky power supply that is not even powered on most of the day?!
After a few days we had a great first version of the product running on the screens in the office. (We started to develop the driver tablets responsible for geo tracking & in general the smart brain in the bus way before this) We used web technologies similar to our consumer web app for the screens (React+Redux) - running on the rasberry Pi that rendered a web page onto the screen. We also considered using Android things which looks very interesting for this sort of thing, however it was in early developer preview by the time we started running our buses so unfortunately we weren't able to use it. The bus lab in our office allowed us to do quick daily iterations on the product side like we are used to from the consumer web app and internal builds of our mobile apps. (At least for the basic UX and UI of things - limited by the fact that you weren't actually on a real bus moving around)
Setup and test the first bus
Once we successfully tested the screen in the office we went to our bus depot and installed the tech in one of our buses. We had help from an electrician to install the power supply, screen and required network cables and then went ahead and installed the router and Raspberry Pi.
We deployed the software to the pi as well as configured the router and were ready to go for a test ride. We did this by using a self-made "poor man's" version of the server automation frameworks Salt & Ansible that we use for our backend machines. The reason for this was mainly the very low disk space available on our 4G routers (i.e. ~20MB) making installing custom software impossible. We deployed to the bus by connecting to the router's wifi and then running the script that would ssh into the router and Pi to set them up accordingly.
Things obviously didn't work smoothly first time around, but we were able to iterate on the product and deploy while on the move and get it working...
How to turn the prototype into a stable service
We have software that works well - but how can we run multiple vehicles reliably for multiple weeks while we keep iterating? When building our backend and web apps for the main Citymapper app we are used to having a reliable infrastructure to build on. We are used to having short lived instances that we treat as being disposable (no single points of failure). If one instance experiences difficulties (e.g. hardware failure) we automatically spin up another one to replace it. That doesn't really work with our buses. Changing the hardware (or losing access to it) means going to the depot to fix it. We will also need to be able to iterate on the software quickly - just like we do with any of our backend, web and mobile apps.
Currently we can only access our bus hardware when physically on the bus - step one is to find a way to remotely access the buses. This would allow us to deploy new versions of the software. In some cases the buses can pull a new version from upstream (e.g. the JS deployment straight from S3), but this would allow use greater control over the process. access logs and debug issues (a centralised logging system would require a lot of bandwidth) * recovery - one of the worst cases would be to have buses with broken systems driving around and not being able to fix them remotely.
However mobile SIM cards do not give you a public IP, making accessing the buses remotely difficult. This restriction makes total sense - as having a public IP would allow anyone to connect. Having a public IP address makes it not only harder to secure access but also exposes you to the noise on the internet altogether. Being exposed to random connection attempts from the internet means that it will take up our mobile bandwidth which is limited. Essentially that would mean that we'd have to pay whenever somebody tries to hack into our bus systems.
Not having remote access wasn't really an option either - especially because we were running a night bus service - any issue with the system would've required us to get out in the early hours of the morning.
Solution: build a VPN network that all buses are connected to
The idea is to have all buses be constantly connect to a VPN network - the buses will initialise the tunnel connection getting around the issue of not being able to make a connection to the buses. Developers would then also be able to connect to the same VPN with their own account and connect to any of the buses that way. Luckily the 4G routers support openVPN. So we spun up another openVPN server (we already had one giving access to some of our internal services from outside the office) Next thing was to create a distinct VPN key for every bus and configure the IP addresses accordingly to ensure we can correctly identify the different buses when connected to the VPN. We now had to configure the routers on the buses which meant one final visit to the bus depot to deploy the buses while having physical access.
This system worked reliably and allowed us to iterate quickly on the software and ship many new software versions during the first weeks of operations while the buses were running.
Should an app company run a bus?
This project was an important moment for us as a company - we transitioned from being 'just' an app company to also run transportation services in the real world. We approached running a bus as a tech company and built our software on top of a completely new 'BaaP' (bus as a platform) system allowing us to rethink how buses are run and perceived by users. Working on this project was also a lot of fun - not every day do you get to play with a bunch of Raspberry Pis in the office or learn how to start and drive a big London bus.
