Data Analytics Environment CBMBT5T07

Syllabus topics (RTMNU)

• Introduction to data analytics
• Types of data - structured, semi-structured and unstructured data
• Types of Data Analytics - Descriptive, Diagnostic, Predictive, Prescriptive
• Importance of data analytics in decision making
• Data analytics lifecycle
• Applications of data analytics in business, healthcare, education and e-commerce
• Overview of data analytics environment

Introduction to Data Analytics

Data means raw facts and figures (sales amounts, names, dates, clicks). Information is processed data that has meaning.

Data analytics is the process of collecting, cleaning, organising and examining data to find useful patterns, trends and insights that help in making decisions.

Example: a supermarket studies its billing data and finds that bread and milk are often bought together, so it keeps them close and offers combo deals.

Features: data-driven; uses tools and software; finds patterns and relationships; supports decisions; can be done on small or very large (big) data.

Types of Data: Structured, Semi-structured and Unstructured

Most of the world's data today is unstructured.

Types of Data Analytics

1. Descriptive analytics – "What happened?" Summarises past data using totals, averages, percentages, charts and reports. Example: monthly sales report showing sales fell 10% in June.
2. Diagnostic analytics – "Why did it happen?" Finds causes by drilling down, comparing and finding correlations. Example: sales fell because a competitor gave a discount and rains reduced footfall.
3. Predictive analytics – "What is likely to happen?" Uses past data, statistics and machine learning to forecast. Example: forecasting Diwali demand; a bank predicting which borrower may default.
4. Prescriptive analytics – "What should we do?" Recommends the best action using optimisation and simulation. Example: an airline app automatically setting fares; route planning for delivery vans.

Importance of Data Analytics in Decision Making

1. Fact-based decisions – reduces guesswork.
2. Understanding customers – their needs, habits and preferences.
3. Cost reduction – finds waste and inefficiency.
4. Better forecasting and planning – stock, staff, finance.
5. Risk management – detects fraud and credit risk.
6. Competitive advantage – faster response to market changes.
7. Performance measurement – tracks KPIs (key performance indicators).
8. New opportunities – new products and markets.

Data Analytics Lifecycle

The lifecycle is the series of stages a data analytics project goes through:

1. Discovery / problem definition – understand the business problem and goals.
2. Data collection – gather data from internal and external sources.
3. Data preparation – clean, integrate and transform data (often the longest stage).
4. Model planning – choose methods and variables.
5. Model building / analysis – apply statistical or machine-learning techniques.
6. Interpretation and communication of results – reports, charts, dashboards.
7. Operationalise / deploy – use the results in daily work and monitor them.

Applications of Data Analytics

• Business: sales forecasting, customer segmentation, pricing, supply chain, fraud detection in banks, credit scoring, HR analytics (attrition prediction).
• Healthcare: predicting disease outbreaks, analysing patient records, hospital bed management, medical image analysis, drug research.
• Education: tracking student performance, identifying students at risk of failing, personalised learning on e-learning apps, course planning.
• E-commerce: product recommendations ("customers also bought"), dynamic pricing, delivery time prediction, review analysis, targeted ads.

Overview of Data Analytics Environment

The data analytics environment is the complete set-up of hardware, software, data, people and processes used to do analytics.

1. Data sources – databases, ERP, websites, sensors, social media.
2. Data storage – databases, data warehouses, data lakes, cloud storage.
3. Processing platforms – Hadoop, Spark, cloud services.
4. Analytics tools – Excel, Python, R, SQL.
5. Visualisation tools – Tableau, Power BI.
6. People – data analysts, data engineers, data scientists, business users.
7. Governance and security – data quality, privacy (Digital Personal Data Protection Act, 2023) and access control.

Likely exam questions

1. (Long, 8/16 marks) What is data analytics? Explain the types of data analytics with examples.
2. (Long, 8/16 marks) Explain the data analytics lifecycle and the importance of data analytics in decision making.
3. (Short, 4 marks) Distinguish between structured, semi-structured and unstructured data.
4. (Short, 4 marks) State any four applications of data analytics in e-commerce or healthcare.
5. (Short, 4 marks) What is a data analytics environment?

Syllabus topics (RTMNU)

• Introduction to analytics tools
• Python environment for data analytics
• Jupyter Notebook and Anaconda environment
• Basic libraries used in analytics such as NumPy and Pandas
• Meaning of R Programming
• Features of R
• R Programming Environment
• Applications of R in Data Analytics
• Introduction to Big Data Platforms — Hadoop: Meaning of Hadoop
• Components of Hadoop
• Advantages of Hadoop
• Apache Spark: Features of Apache Spark
• Applications of Hadoop and Spark

Introduction to Analytics Tools

Analytics tools are software used to collect, clean, analyse and present data. Main groups:

• Spreadsheets – MS Excel, Google Sheets (small data, quick analysis).
• Programming languages – Python, R, SQL.
• Big data platforms – Hadoop, Apache Spark.
• Visualisation/BI tools – Tableau, Power BI.
• Statistical packages – SPSS, SAS.

Python and R are free and open source, so they are widely used by students and companies.

Python Environment for Data Analytics

Python is a free, open-source, high-level programming language created by Guido van Rossum (1991). Its simple, English-like syntax makes it easy for beginners.

Why Python for analytics: easy to learn; huge collection of libraries (NumPy, Pandas, Matplotlib, Scikit-learn); works with databases, Excel and web data; large community; used for machine learning and AI too.

Jupyter Notebook and Anaconda Environment

Anaconda is a free distribution (package) that installs Python, R and hundreds of data science libraries together, with a package manager called conda and a launcher called Anaconda Navigator. It saves beginners from installing each library separately.

Jupyter Notebook is a web-browser-based tool where you write code in cells, run them one by one and see results, tables and charts right below the code. You can also add text notes. Files are saved with the extension .ipynb.

Advantages of Jupyter: interactive, step-by-step working; code, output and explanation in one document; easy to share; good for teaching and reports. Google Colab is a free online Jupyter-like service.

Basic Libraries: NumPy and Pandas

NumPy (Numerical Python) – a library for fast mathematical work on arrays (lists of numbers in one or more dimensions). It provides functions for sum, mean, standard deviation, matrix operations and random numbers.

Pandas – a library for handling tables of data. Its two main structures are Series (one column) and DataFrame (a table with rows and columns, like an Excel sheet). It can read CSV/Excel files, filter rows, handle missing values, group and summarise data.

Other libraries: Matplotlib and Seaborn (charts), Scikit-learn (machine learning).

R Programming: Meaning, Features and Environment

R is a free, open-source programming language and software environment specially designed for statistical computing and graphics. It was developed by Ross Ihaka and Robert Gentleman (University of Auckland).

Features of R:

1. Free and open source.
2. Strong statistical functions – mean, regression, tests, time series.
3. Excellent graphics – plots and charts (ggplot2 package).
4. Thousands of add-on packages on CRAN (Comprehensive R Archive Network).
5. Works on Windows, Mac and Linux.
6. Handles vectors, matrices, data frames and lists.
7. Interpreted language – commands run immediately.

R programming environment: the R console (base R) and RStudio, an integrated development environment with four panes – script editor, console, environment/history, and plots/files/packages/help.

Applications of R: statistical analysis and research, market research and surveys, financial modelling and risk analysis, healthcare and clinical trials, data visualisation, machine learning.

Big Data and Hadoop: Meaning and Components

Big data means data so large, fast or varied that ordinary tools cannot handle it. It is described by the Vs: Volume, Velocity, Variety (and Veracity, Value).

Hadoop is an open-source framework from the Apache Software Foundation for storing and processing big data across clusters of ordinary computers in a distributed way.

Components of Hadoop:

1. HDFS (Hadoop Distributed File System) – stores data by splitting it into blocks spread over many machines, with copies for safety.
2. MapReduce – processing model: the Map step processes data in parallel; the Reduce step combines results.
3. YARN (Yet Another Resource Negotiator) – manages cluster resources and schedules jobs.
4. Hadoop Common – shared libraries and utilities.

Ecosystem tools: Hive (SQL-like queries), Pig, HBase (database), Sqoop.

Advantages of Hadoop: handles huge data; scalable (add more machines); low cost (ordinary hardware, free software); fault tolerant (copies of data); stores all types of data; parallel processing makes it faster.

Apache Spark: Features and Applications of Hadoop and Spark

Apache Spark is an open-source, fast, general-purpose engine for big data processing.

Features: in-memory processing (much faster than MapReduce for many jobs); supports Python (PySpark), Java, Scala, R and SQL; libraries for SQL (Spark SQL), streaming (real-time data), machine learning (MLlib) and graphs (GraphX); can run on Hadoop (YARN, HDFS) or in the cloud; fault tolerant.

Applications: banking fraud detection, telecom call-record analysis, e-commerce recommendations, social media analysis, log analysis of websites, healthcare records, real-time traffic and weather analysis.

Likely exam questions

1. (Long, 8/16 marks) Explain the Python environment for data analytics, including Anaconda, Jupyter Notebook, NumPy and Pandas.
2. (Long, 8/16 marks) What is Hadoop? Explain its components and advantages. Compare Hadoop with Apache Spark.
3. (Short, 4 marks) What is R programming? State its features.
4. (Short, 4 marks) What is Jupyter Notebook?
5. (Short, 4 marks) State the features of Apache Spark.

Syllabus topics (RTMNU)

• Sources of data
• Methods of data collection
• Data preprocessing techniques
• Data cleaning, data integration and data transformation
• Handling missing data and noisy data
• Methods of identify noise in a dataset
• How to Handle Missing Data Values While Data Cleaning
• Data storage formats and basic data management

Sources of Data

Primary data – collected first-hand by the user for a specific purpose (surveys, interviews, experiments). It is original but costly and slow.

Secondary data – already collected by someone else (Census, RBI reports, company annual reports, websites). It is cheap and quick but may not fit the purpose exactly.

Internal sources – company records: sales, billing, accounts, HR, CRM, ERP. External sources – government data (data.gov.in), research agencies, social media, open datasets, purchased data.

Machine-generated data – sensors, IoT devices, website logs, GPS, point-of-sale machines.

Methods of Data Collection

1. Surveys/questionnaires – paper or online (Google Forms).
2. Interviews – personal, telephone or video.
3. Observation – watching behaviour (e.g. counting shop visitors).
4. Experiments – testing changes, e.g. A/B testing of two website designs.
5. Transactional records – billing and online orders.
6. Web scraping and APIs – collecting data from websites and apps.
7. Sensors and IoT – automatic collection.
8. Focus groups – group discussions with selected users.

Data Preprocessing Techniques

Data preprocessing is the process of converting raw, messy data into clean, consistent and usable form before analysis. Raw data is often incomplete, noisy and inconsistent; "garbage in, garbage out" means poor data gives poor results.

Main techniques: (1) data cleaning; (2) data integration; (3) data transformation; (4) data reduction – reducing the size of data while keeping its meaning (sampling, selecting important columns, aggregation).

Data Cleaning, Data Integration and Data Transformation

Data cleaning – finding and correcting errors: filling missing values, removing duplicates, correcting spelling and format errors (e.g. "Nagpur", "nagpur", "NGP" made the same), removing noise and outliers, fixing wrong data types.

Data integration – combining data from several sources into one consistent dataset, e.g. joining sales data from the billing system with customer data from the CRM. Problems: different names for the same field (Cust_ID vs CustomerNo), duplicate records, conflicting values, different units (₹ vs $).

Data transformation – changing data into a suitable form:

• Normalisation – scaling values to a range such as 0 to 1. Min-max formula: (x − min) ÷ (max − min). Example: marks 70, min 40, max 90 → (70 − 40) ÷ (90 − 40) = 0.6.
• Aggregation – daily sales summed into monthly sales.
• Generalisation – age 23 → "20–30" group; city → state.
• Encoding – converting text categories to numbers (Male = 0, Female = 1).
• Attribute construction – creating new columns, e.g. profit = sales − cost.

Handling Missing Data and Noisy Data

Missing data – values that are blank or not recorded (e.g. customer did not give age). Causes: not filled, equipment failure, data entry errors, data lost during transfer.

Noisy data – data with random errors or meaningless values, e.g. age = 250, negative quantity, typing errors.

Handling noisy data:

1. Binning – sort values, divide into bins, and replace values by bin mean, median or boundary (smoothing).
2. Regression – fit values to a regression line to smooth them.
3. Clustering – group similar values; points outside clusters are outliers.
4. Manual inspection combined with computer checks.

Methods of Identifying Noise in a Dataset

• Visual methods – box plots, scatter plots and histograms show unusual points.
• Statistical methods – Z-score (values beyond ±3 standard deviations), and the IQR rule: outlier if below Q1 − 1.5×IQR or above Q3 + 1.5×IQR, where IQR = Q3 − Q1.
• Rule-based checks – valid ranges (age 0–120), formats (10-digit mobile number, PAN pattern), allowed categories.
• Clustering and machine-learning methods for anomalies.
• Duplicate and consistency checks.

Example: Q1 = 20, Q3 = 40 → IQR = 20; limits are 20 − 30 = −10 and 40 + 30 = 70; a value of 95 is an outlier.

How to Handle Missing Data Values While Data Cleaning

1. Delete the record (row) – if few rows are missing; may lose information.
2. Delete the column – if most values in it are missing.
3. Fill manually – practical for small data.
4. Fill with a constant – e.g. "Unknown" or 0.
5. Fill with mean/median/mode – mean for normal numeric data, median when outliers exist, mode for categories.
6. Fill with group mean – average of the same class (e.g. same city).
7. Predict the value – using regression or other models.
8. Forward/backward fill – for time-series data.

Data Storage Formats and Basic Data Management

Common formats: CSV (comma-separated values – simple text tables); Excel (.xlsx); JSON (key–value, used by web apps); XML (tagged data); SQL databases (tables with relations); Parquet (compressed column format for big data); text, image and video files for unstructured data.

Storage systems: relational databases (MySQL, Oracle), NoSQL databases (MongoDB), data warehouses (cleaned, structured data for reports), data lakes (raw data of all types), cloud storage.

Basic data management: proper naming and organisation of files, backup and recovery, access control and security, version control, data quality checks, documentation (data dictionary), and following privacy laws such as the Digital Personal Data Protection Act, 2023.

Likely exam questions

1. (Long, 8/16 marks) What is data preprocessing? Explain data cleaning, data integration and data transformation.
2. (Long, 8/16 marks) What is missing data and noisy data? Explain the methods of identifying noise and of handling missing values.
3. (Short, 4 marks) Distinguish between primary and secondary data.
4. (Short, 4 marks) State any four methods of data collection.
5. (Short, 4 marks) Explain any four data storage formats.

Syllabus topics (RTMNU)

• Concept of data visualization
• Importance of graphical representation of data
• Types of data visualization
• Charts and graphs used in analytics such as bar chart, line chart, pie chart, histogram and scatter plot
• Principles of effective data visualization
• Introduction to visualization tools such as Tableau and Microsoft Power BI
• Basic dashboard concepts
• Components of a dashboard
• Real-world applications of data analytics and case studies

Concept of Data Visualization

Data visualization is the graphical representation of data and information using charts, graphs, maps and dashboards so that patterns, trends and outliers can be seen and understood quickly.

Importance of Graphical Representation of Data

1. Easy to understand – a chart is grasped faster than a table of numbers.
2. Shows trends and patterns – rising or falling sales are seen at once.
3. Comparison – between products, regions, years.
4. Finds outliers and errors.
5. Quick decisions – managers act on dashboards in real time.
6. Better communication and storytelling – reports and presentations become clear and attractive.
7. Handles large data – summarises lakhs of rows in one picture.

Types of Data Visualization

• Comparison – bar and column charts.
• Trend over time – line and area charts.
• Composition (part of whole) – pie, donut and stacked bar charts.
• Distribution – histograms and box plots.
• Relationship – scatter plots and bubble charts.
• Geographical – maps (e.g. state-wise sales map of India).
• Hierarchical – tree maps.
• Tables, heat maps and KPI cards.

Visualizations can also be static (printed charts) or interactive (dashboards with filters and drill-down).

Charts and Graphs Used in Analytics

Pie-chart angle = (value ÷ total) × 360°. Example: brand A sales ₹30 lakh of total ₹120 lakh → 30 ÷ 120 × 360 = 90°.

Principles of Effective Data Visualization

1. Know the audience and purpose – what decision will the chart support?
2. Choose the right chart for the data type.
3. Keep it simple – remove clutter, 3-D effects and unnecessary decoration.
4. Accurate scales – start bar axes at zero; do not distort.
5. Clear titles, labels, legends and units.
6. Use colour carefully – highlight what matters; consistent colours.
7. Highlight key insight – tell a story with the data.
8. Cite the data source.

Visualization Tools: Tableau and Microsoft Power BI

Tableau – a leading visual analytics tool (now owned by Salesforce). Users drag and drop fields to create charts and dashboards. Products include Tableau Desktop, Tableau Cloud/Server and the free Tableau Public. Strengths: beautiful, interactive visuals; connects to many data sources; handles large data.

Microsoft Power BI – Microsoft's business intelligence tool. Components: Power BI Desktop (free, for creating reports), Power BI Service (online sharing) and mobile apps. Uses Power Query to clean data and DAX formulas for calculations. Strengths: works smoothly with Excel and other Microsoft products, low cost, popular in Indian companies.

Basic Dashboard Concepts and Components of a Dashboard

A dashboard is a single screen that displays the most important information (KPIs) using charts and numbers, so a user can monitor performance at a glance – like a car's dashboard.

Types: strategic (top management, long-term KPIs), operational (daily monitoring), analytical (deep analysis of trends).

Components:

1. KPI cards / tiles – big numbers like total sales, profit, number of orders.
2. Charts and graphs – bar, line, pie, maps.
3. Filters and slicers – choose year, region, product.
4. Tables – detailed figures.
5. Title, date and data source.
6. Drill-down and interactivity – click to see details.
7. Alerts/indicators – colours (green/red) showing targets met or missed.

Real-world Applications and Case Studies

• Retail: a supermarket chain uses dashboards of store-wise sales to restock fast-moving items and plan festival offers.
• E-commerce: online marketplaces recommend products using customers' browsing and purchase history.
• Banking: banks use analytics to detect unusual card transactions (fraud) and to score loan applicants.
• Healthcare: during COVID-19, government dashboards showed state-wise cases, vaccination and hospital beds.
• Sports: IPL teams analyse player performance data for auctions and match strategy.
• Transport: ride-hailing and food-delivery apps predict demand and delivery times.

Likely exam questions

1. (Long, 8/16 marks) What is data visualization? Explain its importance and the principles of effective data visualization.
2. (Long, 8/16 marks) What is a dashboard? Explain its components and compare Tableau with Power BI.
3. (Short, 4 marks) Distinguish between a bar chart and a histogram.
4. (Short, 4 marks) What is a scatter plot? When is it used?
5. (Short, 4 marks) State any four types of data visualization.