wynalazkowo - eksperymenty małe i duże

Apache Airflow

Apache Airflow

Recently I had the opportunity to play with this kind of scheduler for data pipelines tasks. It is mega simple to setup either on bare-metal or as docker worker or in Kubernetes using Helmchart acc. to this desciption: https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html This software enables to create data pipelines for extracting data, decorating and saving in different place.Więcej oApache Airflow[…]

Graph database

Graph database

A graph database is defined as a specialized, single-purpose platform for creating and manipulating graphs. Graphs contain nodes, edges, and properties, all of which are used to represent and store data in a way that relational databases are not equipped to do. Graph analytics is another commonly used term, and it refers specifically to the process ofWięcej oGraph database[…]

Zookeeper

Zookeeper

I have heard about this software many times in relation to BigData, Hadoop, Kafka, but never went into detailed knowledge about this until today. So here is short description of this technology and usage of it: ” Zookeeper was invented by Apache as opensource project: https://github.com/apache/zookeeper. ZooKeeper is a centralized service for maintaining configuration information,Więcej oZookeeper[…]