Support for building a data management platform for the Good Design Award

Build a data management platform that enables advanced data analysis and business intelligence to support business decision-making.

Pain points

Data integration between apps within an organization

Background and Issues

The Japan Institute of Design Promotion was operating multiple systems with Good Design Award data, and needed to integrate these data for efficient use. The previous system had the following issues

• Data is scattered across multiple projects, making integration difficult
• Insufficient real-time data processing makes it difficult to obtain data in a timely manner
• It takes time to obtain data and analysis results, making them difficult to utilize in business operations.

Solutions provided

We have built our data management platform using the following technologies

Selection and construction of a data warehouse

• We selected Neo4j, a graph database, as data warehouse to effectively manage complex data relationships and leveraged its flexible schema to build data models.

Building a data pipeline

• Build real-time data streaming and data pipelines using Apache Kafka
• Use Kafka Connect to automate data ingestion into Neo4j from multiple data sources

Data Integration and Conversion

• Transform and aggregate data using Kafka Streams and integrate data into Neo4j in real-time
• Automated ETL processes, such as Debezium and FilePulse, ensure data integrity and consistency

Visualization and Analysis

• Collaboration with projects that use ElasticSearch as the data mart facilitates data retrieval and aggregation

Outcomes and Effectiveness

• Streamline data integration : Data integration from multiple data sources is facilitated, eliminating the problem of data silos.
• Real-time data processing : Real-time data processing with Kafka enables immediate access to the latest business insights to support rapid decision making.
• Improved data visualization and analysis : Utilizing Neo4j's graph database function, data relationships can now be visualized and used as a Single Source of Truth.