With Machine Learning
Structured Query Language is a foundational relational databases language, it is the primary language for manipulating and organizing data in database systems and most familiar to many data practitioners. Spark provides us with SparkSQL which is responsible for executing SQL
Read morePandas is a powerful Python data analysis library that makes working with data easier. It comes with tons of functions for manipulating data. Pandas is a standard data analysis tool in Python. It supports DataFrames as its core data structures.
Read moreSpark DataFrame is a collection of items organized in rows and columns resembling a table in relational database. They are Sparks data structure implemented on top of Sparks Resilient Distributed Datasets (RDDs) and greatly optimized internally. DataFrames can be created
Read moreResilient Distributed Dataset (RDD) is a fault-tolerant collection of elements partitioned across the nodes of the cluster and can be operated on in parallel. RDD is the main abstraction provided by Spark. RDD is created when a file in HDFS
Read moreSpark is a popular open source, distributed and in-memory big data processing engine. It’s loved because of its speed and flexibility to support many programming languages such as Scala, Java, Python and R. Spark is originally written in Scala. The
Read moreWhen it comes to big data storage and processing the tools used are Hadoop and Apache Spark. Hadoop is a distributed storage and processing engine that utilizes MapReduce. One of the limitations of Hadoop is speed of executing big data
Read moreIn today’s world there is more data being generated than ever before. This is due to advancement in technology that has enabled faster processing and transmission of data. Big data is simply data that’s too big to fit in traditional
Read moreData augmentation is a technique of generating extra data with the purpose of improving the performance of machine learning model. Most machine learning algorithms especially neural networks performs well with large and varied sets of data, sometimes the challenge lies
Read moreMore often Data Scientist and Machine learning engineers end up developing models that suffer from data leakage without easily noticing. The model performs perfectly well with high performance on validation set but fail while deployed to production. Data leakage is
Read moreIn this era of big data we often end up with large data that we need to analyse and model. This might take too much time to process, analyse and model. To avoid this we need to select a small
Read more