Singapore is building a Smart Nation. Termed SG100, the vision is to improve the lives of people with more opportunities and stronger communities through a digital economy, government, and society. Citizens and businesses will be empowered through increased access to data. More importantly, they will have the ability to make sense of information to generate the right ideas and solutions.
We know that in the midst of this digital revolution, new waves of technologies will forever transform the way we live and work. One of the anchor initiatives in Singapore is the Smart Nation Sensor Platform (SNSP) led by GovTech. It is a nation-wide sensor platform used to ingest, distribute, contextualise, and act on sensor data, to enable data-driven decisions to improve citizen’s quality of life.
It is certainly an ambitious and complex undertaking with multiple milestones. As a tech for good company that recognises the growing imperative of digitalisation and the importance of collaboration, we are proud to support this journey that involves the co-creation of infrastructure and common technical architecture.
The Trouble with Data Exchange
Many government agencies have a huge quantity of IoT sensors. Take the National Environmental Agency (weather sensors) and Land Transport Authority (traffic sensors) for example. Exchanging data generated between different agencies is a huge undertaking. It is also difficult to manage the massive volume of data generated from the sensors, process it, and then make sense of it.
Furthermore, the exchange of data between sensors is also regulated by certain protocols. The big challenge here is security. To establish a secure connection and validate credentials, government agencies would have to go through the arduous process of contacting each other to obtain an SSL certificate before they can access sensor data.
Adopting a co-development approach, we worked with GovTech in a joint project to create a Sensor Data Exchange (SDX) platform that can integrate myriad forms of sensor data for seamless exchange, management, and interpretation.
Introducing the One-Stop Shop
SDX leverages open-source and cloud services. It is an unified, scalable, and extensible platform that encompasses public IoT sensors. Government agencies can now publish and subscribe to real-time data using a self-service portal both on-premise and in the commercial cloud.
By co-creating data-rich services for planning and operations, government agencies can work together more effectively to improve our quality of life and unlock new possibilities. Government agencies could publish and subscribe to each other’s real-time on the SDX application. For example, an ambulance would be able to discern the fastest route to the hospital in an emergency situation.
How Did We Do It?
Here is where we get technical. On the technology front, we used:
- Angular (frontend),
- Golang (backend),
- RabbitMQ,
- GCC-AWS,
- Terraform
The middleware uses a publish and subscribe model, where senders of messages are called publishers and receivers are called subscribers. Publishers are not required to program published messages to be sent directly to specific subscribers, but instead characterise them into topics. Subscribers who have expressed interest and requested to receive messages in one or more topics can then receive them once their request has been approved.
Fine-grained quality of service (QOS) and automatic discovery will be enabled. Software-defined abstraction layering also ensures that sensor data will be managed in a centralised system with built-in security so that users will not or need to have access to physical sensors.
As simple as we’ve made it seem, putting everything together behind the scenes was not as easy as it sounds.
Navigating Roadblocks to Success
Marking the start of this innovation partnership, we embedded ourselves into the SDX engineering team and worked in close collaboration with their engineering lead. The first task the team reviewed was working with legacy systems. We needed to work out how to build an application logic that is different from conventional implementation. Transitioning from old features to new features and developing change of functions was an issue with no prior use cases or scenarios to learn from. This was technically challenging from a developer’s perspective.
Then we needed to define the project’s aim. This project had to optimise and monitor the core engine's behavior to achieve a high rate of request per second. However, we did manage to set up monitoring and found solutions to optimise the performance.
Each sensor was connected in a secure way with control permissions over a topic that can be published or subscribed. We used Elasticsearch as a search engine to ensure search performance was good given the volume of data. For monitoring, we used Splunk for security-related reports, ELK to monitor application logs, and Prometheus/Grafana for visualising application performance metrics.
Lastly, the DevOps onboarding process was also tedious with many moving components. To mitigate our roadblocks and enhance the team’s practices, we:
- Began with version control by defining proper git-flow and branching strategy
- Used Bamboo for continuous integration & continuous delivery
- Ensured continuous integration and continuous delivery pipelines with integration to different scanning tools like SonarQube, NexusIQ, and HP Fortify
- Built notifications in every stage of the pipeline
A good practice when hosting the application in a public cloud is to follow strict security guidelines. We followed CIS benchmark set of rules in order to access the security related loops.
Laying the Foundations
Despite the challenges of remote working, our team managed to go live on the first phase (SDX 1.0) of our roadmap without any complications, and successfully onboarded a few government agencies to the application (SDX 2.0).
One of the key highlights was also the implementation of blue-green deployment by taking advantage of swapping autoscaling groups in a cost-effective way and with zero downtime.
Along the way, we learned a lot about new tools and services, including Terraform, Bamboo, and a variety of GCC-AWS APIs. Projects like these are a great place to experiment with new technology. The flexibility to choose what best worked for us and presenting several options to the discussion table quickly resulted in the creation of stories that could bring the most value to our client.
The journey doesn't end here - the work we have completed is a piece of an unfinished and exciting puzzle. With the building blocks in place, we are exploring to expand SDX and its capabilities beyond just the public sector.
Turning Ideas Into Reality
In becoming a Smart Nation with endless opportunities enabled by technology, citizens and businesses need to have access to information that allows them to make more meaningful choices to improve their quality of life.
SNSP will enable government agencies to facilitate better sense-making by collecting, analysing, and sharing data to people in a cost-effective manner. Shaping the future one step at a time, our work on the SDX project brings us closer to making this happen.
