Data analysis with Pandas, WebDataRocks and Datapane

Reading time:   2 min

data-analysis-with-pandas-webdatarocks-and-datapane

Anfavea, the National Association of Motor Vehicle Manufacturers in Brazil, discloses in its website some monthly statistics about the production and licensing of vehicles. People in the automotive market who use these data for analysis and forecast, certainly encountered various limitations in the Excel file available for download: the report looks like a PowerPoint presentation, with indented cells, values referring to subtotals in the middle of the lines and other small details that make it very difficult to use.

anfavea-excel



To make these data useful, I developed a Python script with a powerful analysis tool called Pandas, in order to be able to clean the original data and create a new Excel or CSV file.

The code is available on my Github repository.

OK, so now we have a new Excel file; but what about the possibility to create a Pivot Table and render it on this webpage to be able to filter the data, for example by segments (cars or trucks) ?

No problem, we can use the Free Web Pivot Table Tool from WebDataRocks to visualize it.

The code is very simple: as explained in their quick start guide, we define some Javascript to parse the data from the CSV file that we generated with our Python code and we render it.

<link href="https://cdn.webdatarocks.com/latest/webdatarocks.min.css" rel="stylesheet"/>
<script src="https://cdn.webdatarocks.com/latest/webdatarocks.toolbar.min.js"></script>
<script src="https://cdn.webdatarocks.com/latest/webdatarocks.js"></script>
<script>
var pivot = new WebDataRocks({
	container: "#wdr-component",
	toolbar: true,
    report: "/images/anfavea-report.json"
});
</script>

This is the result:

1
1
2
3
4
2
3
4
5
6
7
8
9
10
11
12
13
Total Sum of Jan
Total Sum of Fev
Total Sum of Mar
Total Sum of Abr
Total Sum of Mai
Total Sum of Jun
Total Sum of Jul
Total Sum of Ago
Total Sum of Set
Total Sum of Out
Total Sum of Nov
Total Sum of Dez
5
Automóveis
6
Caminhões
7
Comerciais leves
8
Ônibus (chassi)
9
Grand Total
10
11
12
13
14
92 238
100 032
108 272
109 707
139 483
133 781
135 431
0
0
0
0
0
8 705
8 091
10 056
9 384
10 394
10 975
11 554
0
0
0
0
0
24 536
23 287
27 129
27 024
35 731
31 907
33 766
0
0
0
0
0
1 039
921
1 362
1 127
1 456
1 404
1 243
0
0
0
0
0
126 518
132 331
146 819
147 242
187 064
178 067
181 994
0
0
0
0
0
Segmento
Associada
Grupo
Marca
Created using

And what about the possibility to execute the script, obtain some reports and be able to interact with it (for example setting the year of the analysis) ?

I discovered that there is a tool that allows you to do that (Datapane), but it is no longer actively maintained.

Thomas Cenni

Sounds interesting ?

Contact me to discuss your need.

Thomas Cenni
Thomas Cenni is an Electronic Engineer with more than 20 years of experience in program management and software engineering. Passionate about software engineering and development, experienced leader with a strong background in the end-to-end lifecycle of product development, he is also a certified SAFe® 6 agilist. Entrepreneur, credible and highly motivated, he showed a great ability to adapt to various cultures and countries, assuming different roles and challenges in Italy, Brazil and France. He is fluent in English, French, Italian and Brazilian Portuguese.