15-110: Principles of Computing - Week 10

Introduction to Data Analysis

• Data, information, and knowledge
• Data: collection of recorded facts

  
  Example: Supermarket transactions
  transaction1: milk, bread, chips, coke, ..., $30.45
  transaction2: broccoli, milk, chicken, rice, ..., $20.78
  transaction3: chicken, chips, milk, coke, ..., $25.10
  transaction4: milk, broccoli, salmon, pasta, ..., $103.15
  transaction5: chips, coke, milk, beef, ..., $40.89
  transaction6: salmon, milk, carrots, lettuce, ..., $120.32
  transaction7: eggs, milk, coke, chips, ..., $35.98
  

• Information: patterns and structure underlying the data

  
  Example from previous supermarket transactions:
  - Milk is a very frequent item.
  - Chips and coke are usually bought together.
  - Salmon and chips + coke are not usually bought togeter.
  - Salmon and vegetables are usually bought together.
  - Transactions with salmon usually total more than $100.
  

• Knowledge: organized and interconnected information that enables humans to make decisions and take actions

  
  Example:
  - Let's have a discount on milk to attract many more customer to our store.
  - Let's raise the price of milk because people would buy it anyway.
  - Let's put chips and coke next to each other on the shelf, and put other snacks next to them.
  - Let's have a promotion on salmon and increase the price of vegetables.
  

• From raw data to knowledge
1. Data collection: surveys, log files, web pages, ...
2. Data preparation and cleaning: ensure consistency, formatting, deal with outliers
3. Pattern dicovery and model creation: manually, computationally (statistics, data mining)
4. Model evaluation and validation: is our model sound? Is it general enough?
5. Model interpretation: what does the model mean? How can it inform our decisions?