Navigate back to the homepage

How to create a COVID dashboard web application with Python

And, how we attracted thousands of users, and received contributions from developers around the world.
Brian Ruiz
· 5 min read
0

Why start this project?

When I started this side project there were not too many dashboards or site to track data for the Coronavirus pandemic, other than the very popular web app creaeted by John Hopkins University. My idea was to create an open-source version of that which encouraged collaboration to derive some creative design and functional ideas. And, that’s precisely what this project became.

COVID-19 is an infectious disease caused by a newly discovered coronavirus. The best way to prevent and slow down transmission is be well informed. —who.int (05, 2020)

We have contributors from Spain, Poland, India, and various places in the U.S. who came together to create something useful and meaningful. Furthermore, the site has accumulated over 10k visits as shown by web analytics. And, the project is still being maintained and improved upon, thanks to the help of developers.

unsplash
Coronavirus protected dude

Where the data comes from

A large portion of the data we comes from one of the original data repositories tracking COVID-19 cases, Johns Hopkins University: CSSE 2019-ncov data repository. Additional data collected from Our World in Data: OWID’s GitHub data repository. Lastly, to fill in some requirements we retrieved data for daily cases from New York Times COVID GitHub data repository.

For this project it was crucial to have accurate, reliable and trustworthy datasets. Which explains why the multiple aforementioned sources were diligently picked. Having not done so would only lead to misinformation and inaccurate representations of the global pandemic, which could have it’s own repercussions at scale.

Technology stack

This project began from Python based Jupyter notebooks, and the use of Plotly, “the leading front-end for ML & data science models in Python, R, and Julia.” Django. But I needed a way for this to be accessible to the public therefore I started using the Django web-framework, Django is a high-level Web framework that encourages rapid development and clean, pragmatic design. To bootstrap the initial dashboard grid we used Appseed and Bootstrap 4 to fill-in where we needed custom UI elements.

Several times I reconsidered the deployment of this project, but ultimately decided to deploy it to Heroku. Heroku is a cloud-based platform that offers a easy way to deploy your own Django-based web applications. Though, other contendors included elastic beanstalk (AWS) and Google App Engine, Heroku included tools which seemed much more valuable at a free tier. Furthermore, Heroku provides seamless CI and made it super easy to test pull-requests from other contributors.

technology used in project

Front-end design

Overall the general approach of the application is to display a grid of different cards. The cards represent different form of data organized into a it’s respective way, and the user can interact with the data in a number of ways. For instance we have a map card which uses COVID case data stacked on top of GPS coordinates. In another card we have plot which shows the overall trajectory of confirmed cases using time-series data with dates.

And, it was important to design the grid in such way that the cards of data were responsive and friendly to different devices and viewports. The following image exemplifies that.

interactive UI
Responsive web interface

How it works

The following is a snippet on how we’re using Python and Pandas to pull the data from different sources and returning that data to the front-end using AJAX asynchronous requests. Using AJAX requests allows us to load the data asynchronously, which is a great way to improve the user experience by significantly reducing the initial page load time. Note that code may be incomplete for concision.

1. Web scraping with Pandas 🐼

Here we defined different function that read from a specific URL containing a CSV file of raw data, convert it to a Pandas dataframe, and return the dataframe.

1import pandas as pd
2
3def confirmed_report():
4 # Returns time series version of total cases confirmed globally
5 df = pd.read_csv('https://raw.githubusercontent.CSSEGISandData/...csv')
6 return df
7
8def deaths_report():
9 ...

2. Data cleaning & formatting 🧼

Now we define function realtime_growth which cleans and manipulates the dataframe returned from the above function. We need to do this because the data is not in the format we want, and we need to clean it up.

1def realtime_growth():
2 df1 = confirmed_report()[confirmed_report().columns[4:]].sum()
3 df2 = deaths_report()[deaths_report().columns[4:]].sum()
4 df3 = recovered_report()[recovered_report().columns[4:]].sum()
5
6 growth_df = pd.DataFrame([])
7 growth_df['Confirmed'], growth_df['Deaths'], growth_df['Recovered'] = df1, df2, df3
8 growth_df.index = growth_df.index.rename('Date')
9
10 yesterday = pd.Timestamp('now').date() - pd.Timedelta(days=1)
11
12 return growth_df

3. Sending AJAX request 📨

Using AJAX in javascript to send an asynchronous request to the Python backend. If successful then we create the data visualization —in this case, a scatter plot with visible line traces— using plotly.js.

1function load_realtime_growth_chart() {
2 var xhttp = new XMLHttpRequest();
3 xhttp.onreadystatechange = function() {
4 if (this.readyState == 4 && this.status == 200) {
5 /* plotly.js code here*/
6 }
7 };
8 xhttp.open("GET", "realtime_growth");
9 xhttp.send();
10}

4. Returning HTTP response 🪃

This function will now return the HTTP response. Whenever it is requested by the front-end, it will return the dataframe which is used by Plotly to create the plot.

1def realtime_growth(request):
2 df = realtime_growth();
3 df.index = pd.to_datetime(df.index)
4 df.index = df.index.strftime('%Y-%m-%d')
5 return HttpResponse(df.to_json(orient='columns'), content_type='application/json')

5. Plotly to visualize the data 📊

We then create the D3.js-based line and scatter plots in JavaScript using Plotly. This creates an interactive, scientific data visualization easily on the Web browser. This one will probably be the most complex looking part of the code, because it takes a lot of tweaking to get the plot to your desired look and feel. But trust me it’s simpler than it looks. www.plot.ly has great documentation and examples.

1var plot_data = [confirmed_trace, recovered_trace, deaths_trace];
2var plot_layout = {
3 paper_bgcolor:'rgba(0,0,0,0)',
4 plot_bgcolor:'rgba(0,0,0,0)',
5 yaxis: {automargin: true, type: "log", gridcolor: "#32325d"},
6 xaxis: {automargin: true, showgrid: false},
7 showlegend: false,
8 font: {color: '#ced4da'},
9 margin: {t:0, l:0, r:0, b:0},
10 hovermode: "closest",
11 updatemenus: [
12 {
13 visible: true,
14 type: "dropdown",
15 buttons: [
16 {method: "relayout", label: "Logarithmic", args: [{"yaxis.type": "log"}]},
17 {method: "relayout", label: "Linear", args: [{"yaxis.type": "linear"}]}
18 ],
19 x: 0.05,
20 xanchor: "auto",
21 bgcolor: "#6236FF",
22 bordercolor: "rgba(0,0,0,0)"
23 }
24 ]
25 };
26);

The Final Product

In the end, we have a fully functional real-time data visualization that is responsive to different device viewports and fully interactive to both mouse and touch controls. All in the most effective contentdelivery system that exists, the internet.

Mobile  UI
Dashboard view on a mobile device

Conclusion

So did I achieve my initial goal? Yes, exceedingly, I believe. There is a tremendous amount of appreciation for this project knowing that tens of thousands of users have even opened the website and interacted with it. And, the idea that some of those users left the site learning a little more about the COVID-19 pandemic is rewarding.

If you’re a developer and wish to contribute to the ongoing project you may checkout the more in-depth documentation on Getting Started section from the Github repository. There’s also a Gitter based chat if you have any questions or conerns.

More articles from BR Portfolio

How we managed and monitored 100k shipments per week

Industry leading logistics through proprietary in-house developed tracking systems

· 1 min read

How we created a data analytics platform to maximize profits

Used by fortune 500 clients, we developed a data monitoring application which optimizes asset management.

· 2 min read
Link to $https://github.com/BrianRuizyLink to $https://www.linkedin.com/in/brianruizy/Link to $https://www.instagram.com/brianruizy/Link to $mailto:brianruiz@protonmail.com