Vivid DB is a Python based universal data mapper and SQL-Database engine, that implements high-performance and security requirements for large-scale data analysis applications. The primary goal of Vivid DB is to separate data integration and data analysis into independent tasks.
To achieve this goal, Vivid DB implements the two fundamental layers of a data warehouse, the integration layer and the staging layer:
- The integration layer utilizes SQLAlchemy to allow it's connection to a variety of SQL-Databases (e.g. IBM DB2, Oracle, SAP, MS-SQL, MS-Access, Firebird, Sybase, MySQL, Postgresql, SQLite, etc.). Thereupon it provides native support for flat file databases (e.g. CSV-Tables, R-Table exports), laboratory measurements and data generators.
- The staging layer is implemented as a native SQL-Database engine with full SQL:2016 support, vertical data layout and real-time encryption. This allows the data analysis application to integrate a variety of different data sources, by using a unified data interface (and SQL dialect).
Installation using PIP
$ pip install vivid-db
Online analytical processing and predictive analytics in combination with machine learning provides a new challenge in data-warehousing: The response time for large transactions of data from different domains. ... read more
For the current development- and exploration process in data science three obstacles in particular appear as outstanding hurdles, when it comes up to realize projects - and even more, when it comes up to venture collaborations. ... read more