With Machine Learning
Feature scaling is a technique of normalizing or standardizing data into a certain range suitable for fitting a machine learning algorithm. If we don’t scale our data for example we have a variable called age with values in range 12
Read moreOne key challenge with most of the machine learning algorithm is the inability to work with categorical variables. They need the features to be converted to numeric form. The processing of transforming categorical features to numerical form is referred to
Read moreFeature selection is a feature engineering process (sometimes considered as data pre-processing method) of choosing the variables with high predictive power from the dataset. It can be conducted for both supervised and unsupervised learning. For supervised approach we statistically evaluate
Read moreWorking with data with high dimension often possess a challenge commonly referred to as the “curse of dimensionality”. It’s difficult to analyse, visualize and model high-dimensional data hence the need to transform the data from high-dimension to low-dimension. The process
Read moreFeature engineering is the process of using domain knowledge and scientific techniques to select and transform the most important features/variables that will yield an optimum machine learning or statistical model. The main goal of feature engineering is to increase the
Read moreStructured Query Language is a foundational relational databases language, it is the primary language for manipulating and organizing data in database systems and most familiar to many data practitioners. Spark provides us with SparkSQL which is responsible for executing SQL
Read morePandas is a powerful Python data analysis library that makes working with data easier. It comes with tons of functions for manipulating data. Pandas is a standard data analysis tool in Python. It supports DataFrames as its core data structures.
Read moreSpark DataFrame is a collection of items organized in rows and columns resembling a table in relational database. They are Sparks data structure implemented on top of Sparks Resilient Distributed Datasets (RDDs) and greatly optimized internally. DataFrames can be created
Read moreResilient Distributed Dataset (RDD) is a fault-tolerant collection of elements partitioned across the nodes of the cluster and can be operated on in parallel. RDD is the main abstraction provided by Spark. RDD is created when a file in HDFS
Read moreSpark is a popular open source, distributed and in-memory big data processing engine. It’s loved because of its speed and flexibility to support many programming languages such as Scala, Java, Python and R. Spark is originally written in Scala. The
Read more