TDS Notes

1. File Formats & Data Handling

Mon, 01 Jan 2024 00:00:00 +0000

1. File Formats & Data Handling#

1. Markdown (`.md`)#

Lightweight, human-readable documentation format
Converts easily to HTML
Simple syntax:

# Heading 1
## Heading 2
**bold**, *italic*, `code`
- bullet point
[link text](url)
![image](path)

Best for: README files, documentation, notebooks, wikis
❌ NOT suitable for complex formatted documents (use DOCX)
❌ NOT binary - fully readable as plain text

2. DOCX (Word Document)#

Microsoft Word format
Supports rich formatting: tables, headers, page numbers, styles
NOT ideal for lightweight documentation
NOT easily converted to HTML without tools
Best for: formal reports, letters, documents sent to clients
❌ Binary format - not human-readable as plain text
❌ Requires Word or compatible software to open properly

3. Binary Blob#

Raw binary data with no human-readable structure
Cannot be opened in text editor meaningfully
Examples: compiled files, encrypted data, raw image data
❌ NOT suitable for documentation
❌ NOT human-readable

4. Parquet#

Columnar storage format
Most efficient for large numeric arrays
Minimal storage overhead - uses compression
Stores data type information (schema-aware)
Best for: big data, analytics pipelines, data lakes
Comparison:

Format	Human-Readable	Efficient	Schema	Best For
Parquet	❌	✅✅✅	✅	Large numeric arrays
CSV	✅	❌	❌	Sharing with stakeholders
JSON	✅	❌	Partial	APIs, nested data
XML	✅	❌❌	✅	Legacy systems

# Reading/writing Parquet with pandas
df.to_parquet('data.parquet')
df = pd.read_parquet('data.parquet')

5. XML (eXtensible Markup Language)#

Human-readable but very verbose
Uses tags: <tag>value</tag>
High overhead - lots of repeated tag text
NOT efficient for large numeric data
Best for: configuration files, legacy data exchange, SOAP APIs

<!-- Example: verbose compared to other formats -->
<record>
 <name>John</name>
 <age>25</age>
 <salary>50000</salary>
</record>

6. JSON (JavaScript Object Notation)#

Human-readable, flexible format
Supports nested structures
NOT efficient for large numeric arrays
Best for: APIs, configuration files, nested/hierarchical data

import json

# Reading JSON
with open('data.json') as f:
 data = json.load(f)

# Writing JSON
with open('output.json', 'w') as f:
 json.dump(data, f, indent=2)

# Parse JSON string
data = json.loads('{"name": "John", "age": 25}')

# Convert to JSON string
json_str = json.dumps(data, indent=2)

# From API response
response = requests.get(url)
data = response.json() # parse JSON response directly

7. CSV (Comma-Separated Values)#

Simple, human-readable tabular format
Best for sharing results with non-technical stakeholders
Can be opened directly in Excel
No schema enforcement - all values stored as text
Limitations: no data types, no nested structures, no compression

import pandas as pd
import csv

# pandas
df = pd.read_csv('data.csv')
df.to_csv('output.csv', index=False)

# Custom delimiter
df = pd.read_csv('data.tsv', sep='\t') # tab-delimited

# Built-in csv module
with open('data.csv') as f:
 reader = csv.DictReader(f)
 for row in reader:
 print(row)

✅ Best for sharing summary results with non-technical stakeholders
❌ Python pickle (.pkl) - not human-readable, not cross-platform safe
❌ JSON with nested structures - harder to open in Excel
❌ Binary database file - not accessible without tools

8. File Format Comparison - Complete#

Format	Readable	Size	Types	Nested	Best Use
Markdown	✅	Small	Text	❌	Documentation
CSV	✅	Medium	No	❌	Stakeholder sharing
JSON	✅	Medium	Partial	✅	APIs, config
XML	✅	Large	✅	✅	Legacy, SOAP
Parquet	❌	Small	✅	✅	Big data, analytics
DOCX	❌	Medium	✅	❌	Formal documents
Pickle	❌	Small	✅	✅	Python-only objects
HDF5	❌	Small	✅	✅	Scientific data

9. Excel Text-to-Columns - Auto-Date Conversion Problem#

Problem:#

Tab-delimited file contains: "3/10"
Excel auto-interprets as: "March 10" (date)

Fix:#

✅ Format the column as Text BEFORE applying Text-to-Columns
This tells Excel to treat the column as plain text, not numbers or dates

Steps:#

1. Select the column(s) in Excel
2. Format → Cells → Text
3. THEN apply Data → Text to Columns
4. In the wizard → select "Text" as column data format

❌ Changing delimiter to commas → doesn’t fix data type issue
❌ Using General format → still auto-converts to date
❌ Manually changing cells after → tedious, error-prone

10. Excel Text-to-Columns - Leading Zeros Problem#

Problem:#

File contains: "001234"
Excel auto-converts to: 1234 (drops leading zeros)

Fix:#

✅ Format cells as Text BEFORE using Text-to-Columns
Same fix as above - Text format preserves leading zeros

Common cases where leading zeros matter:#

Postal/ZIP codes: 00123
Phone numbers: 0412345678
Product codes: 001234
Student IDs: 0045

11. The Universal Fix - Format as Text First#

When to ALWAYS format as Text before import:#

Values that look like dates but aren’t (3/10, 1-2, 12/5)
Values with leading zeros (001234, 0045)
Long numeric IDs that might be converted to scientific notation
Values with special characters that might be misinterpreted

In Text-to-Columns Wizard:#

Step 3 of 3 in wizard:
→ Click on the problematic column
→ Under "Column data format" select "Text"
→ Finish

12. Python `csv` Module#

import csv

# Reading CSV
with open('data.csv', 'r', newline='') as f:
 reader = csv.reader(f)
 header = next(reader) # skip header
 for row in reader:
 print(row) # list of values

# Reading as dictionary
with open('data.csv', 'r') as f:
 reader = csv.DictReader(f)
 for row in reader:
 print(row['name'], row['age'])

# Writing CSV
with open('output.csv', 'w', newline='') as f:
 writer = csv.writer(f)
 writer.writerow(['name', 'age', 'salary']) # header
 writer.writerow(['John', 25, 50000])

# Custom delimiter (tab-delimited)
with open('data.tsv', 'r') as f:
 reader = csv.reader(f, delimiter='\t')

# Handling special characters
with open('data.csv', 'r', encoding='utf-8') as f:
 reader = csv.reader(f)

13. Python `json` Module - Complete Reference#

import json

# Parse JSON string → Python object
data = json.loads('{"name": "John", "age": 25}')

# Convert Python object → JSON string
json_str = json.dumps(data, indent=2)

# Read JSON file → Python object
with open('data.json', 'r') as f:
 data = json.load(f)

# Write Python object → JSON file
with open('output.json', 'w') as f:
 json.dump(data, f, indent=2)

# Handling nested JSON (API response)
response_json = {
 "weather": [{"description": "clear sky"}],
 "main": {"temp": 298.15, "humidity": 60}
}

# Access nested fields
description = response_json['weather'][0]['description'] # ✅
temperature = response_json['main']['temp']

# Common json.dumps options
json.dumps(data, indent=2) # pretty print
json.dumps(data, sort_keys=True) # sort keys alphabetically
json.dumps(data, ensure_ascii=False) # allow non-ASCII characters
json.dumps(data, default=str) # handle non-serializable types

14. Python Pickle - When NOT to Use#

import pickle

# Save Python object
with open('model.pkl', 'wb') as f:
 pickle.dump(model, f)

# Load Python object
with open('model.pkl', 'rb') as f:
 model = pickle.load(f)

Pickle Limitations:#

❌ NOT human-readable (binary format)
❌ NOT cross-platform safe (Python version dependent)
❌ NOT secure - can execute arbitrary code on load
❌ NOT suitable for sharing with non-technical stakeholders
✅ OK for: saving Python ML models temporarily
✅ Better alternatives: joblib for ML models, parquet for data, json for configs

Quick Reference Card#

Documentation needed?
→ Markdown (.md)

Sharing data with non-technical users?
→ CSV (opens in Excel)

Large numeric dataset in pipeline?
→ Parquet

API data / nested data?
→ JSON

Formal Word document?
→ DOCX

Excel importing numbers as dates or dropping leading zeros?
→ Format column as Text BEFORE Text-to-Columns

Comparing formats for efficiency?
→ Parquet > CSV > JSON > XML
 (storage efficiency, NOT readability)

2. Shell & Command Line

Tue, 02 Jan 2024 00:00:00 +0000

2. Shell & Command Line#

1. `ls` - List Directory Contents#

Basic Usage:#

ls # list files in current directory
ls /path/to/dir # list files in specific directory

All Important Flags:#

ls -a # list ALL files including hidden ✅ (exam answer)
ls -l # long format (permissions, size, date, owner)
ls -h # human-readable sizes (KB, MB, GB) - NOT hidden files ❌
ls -la # long format + hidden files combined
ls -lh # long format + human-readable sizes
ls -lt # sort by modification time
ls -lr # reverse order
ls -lS # sort by file size
ls -R # recursive (list subdirectories too)

Hidden Files:#

Start with . (dot)
Examples: .gitignore, .env, .bashrc, .ssh/
ls without -a → hidden files NOT shown
ls -a → ALL files shown including hidden ✅
❌ ls -h → human-readable sizes (NOT hidden)
❌ show -a → not a valid Unix command
❌ dir /ah → Windows command only

Long Format (`ls -la`) Output:#

ls -la
# Output:
# drwxr-xr-x 5 user group 4096 Dec 14 16:45 .
# drwxr-xr-x 20 user group 4096 Dec 13 10:22 ..
# -rw-r--r-- 1 user group 220 Dec 14 09:00 .bashrc ← hidden
# -rw-r--r-- 1 user group 156 Dec 14 09:00 .gitignore ← hidden
# drwxr-xr-x 2 user group 4096 Dec 14 16:45 data/
# -rw-r--r-- 1 user group 2048 Dec 14 16:45 script.py
#
# Columns: permissions | links | owner | group | size | date | name

2. `find` - Search for Files by NAME#

# Find files by name
find . -name "*.py" # all Python files
find . -name "data.csv" # specific file
find /home -name "*.log" # in specific directory
find . -name "Pine Ridge" # ❌ finds FILE named "Pine Ridge"
 # NOT content inside files

# Find by type
find . -type f -name "*.csv" # files only
find . -type d -name "data" # directories only

# Find by size
find . -size +10M # files larger than 10MB
find . -size -1k # files smaller than 1KB

# Find and execute command
find . -name "*.log" -exec cat {} \; # cat all log files
find . -name "*.pyc" -exec rm {} \; # delete all .pyc files

# Find modified recently
find . -mtime -7 # modified in last 7 days
find . -mtime +30 # modified more than 30 days ago

`find` vs `grep`:#

Command	Searches	Example
`find`	File NAMES	`find . -name "*.log"`
`grep`	File CONTENTS	`grep "error" logfile.txt`

3. `cat` - Display File Contents#

cat file.txt # display entire file
cat file1.txt file2.txt # display multiple files
cat -n file.txt # display with line numbers
cat > newfile.txt # create new file (type content, Ctrl+D to end)
cat >> file.txt # append to file

❌ Does NOT search content (use grep)
❌ NOT suitable for very large files (use less or head)

4. `grep` - Search Text INSIDE Files#

Basic Usage:#

grep "search_term" file.txt # search in file ✅
grep "Pine Ridge" evacuation.txt # find all lines with "Pine Ridge"
grep "error" app.log # find error lines in log

Important Flags:#

grep -i "term" file.txt # case-insensitive search
grep -n "term" file.txt # show line numbers
grep -v "term" file.txt # invert - lines NOT containing term
grep -c "term" file.txt # count matching lines
grep -l "term" *.txt # list files containing term
grep -r "term" ./dir/ # recursive search in directory
grep -w "term" file.txt # whole word match only
grep -A 2 "term" file.txt # show 2 lines AFTER match
grep -B 2 "term" file.txt # show 2 lines BEFORE match
grep -E "pattern" file.txt # extended regex ✅ (exam relevant)

grep with Regex:#

grep -E "[0-9]+" file.txt # lines with numbers
grep -E "^ERROR" file.txt # lines starting with ERROR
grep -E "\.py$" file.txt # lines ending with .py
grep -E "(GET|POST)" access.log # lines with GET or POST
grep -E "1[0-5][0-9]" file.txt # match time range pattern

5. `awk` - Field Extraction & Processing#

Basic Usage:#

awk '{print $1}' file.txt # print first field
awk '{print $3, $7}' file.txt # print 3rd and 7th fields ✅
awk -F',' '{print $2}' file.csv # use comma as delimiter
awk '{print NR, $0}' file.txt # print with line numbers

Common Patterns:#

# Print specific fields from log
awk '{print $1, $4, $7}' access.log

# Filter and print
awk '$9 == 200 {print $0}' access.log # lines where 9th field = 200

# Calculate sum
awk '{sum += $5} END {print sum}' file.txt

# Print lines matching condition
awk '$7 ~ /\/checkout/' access.log # URL contains /checkout/

# Count occurrences
awk '{count[$7]++} END {for (url in count) print count[url], url}' log

Special Variables:#

Variable	Meaning
`$0`	Entire line
`$1`, `$2`…	Field 1, 2…
`NR`	Current line number
`NF`	Number of fields
`FS`	Field separator

6. `sort` - Sort Output#

sort file.txt # alphabetical sort
sort -n file.txt # numeric sort
sort -r file.txt # reverse order
sort -nr file.txt # numeric reverse ✅ (exam relevant - sort -nr)
sort -k2 file.txt # sort by 2nd field
sort -k2 -n file.txt # sort by 2nd field numerically
sort -u file.txt # sort and remove duplicates
sort -t',' -k2 file.csv # sort CSV by 2nd column

7. `uniq` - Count/Remove Duplicates#

uniq file.txt # remove consecutive duplicates
uniq -c file.txt # count occurrences ✅ (exam relevant)
uniq -d file.txt # show only duplicates
uniq -u file.txt # show only unique lines

# IMPORTANT: uniq only works on SORTED input
# Always sort before uniq:
sort file.txt | uniq -c

8. `head` and `tail` - View File Portions#

head file.txt # first 10 lines (default)
head -n 20 file.txt # first 20 lines
head -1000 file.txt # first 1000 lines

tail file.txt # last 10 lines
tail -n 20 file.txt # last 20 lines
tail -f file.txt # follow file in real-time (for logs) ✅

9. `wc` - Word/Line/Character Count#

wc file.txt # lines, words, characters
wc -l file.txt # count lines only ✅
wc -w file.txt # count words only
wc -c file.txt # count bytes
wc -m file.txt # count characters

# Count lines matching pattern
grep "ERROR" app.log | wc -l

10. Shell Pipeline - Combining Commands#

The `|` (pipe) operator:#

Takes output of left command as input to right command
Can chain multiple commands together

Complete Log Analysis Pipeline:#

# ✅ Exam-relevant pipeline for large log file analysis:
grep -E "(0[89]|1[0-8]):[0-5][0-9]" traffic.log \
| awk '{print $3, $7}' \
| sort \
| uniq -c \
| sort -nr

# Breakdown:
# grep -E "..." → filter lines matching time pattern
# awk '{...}' → extract fields 3 and 7
# sort → sort alphabetically (needed before uniq)
# uniq -c → count unique occurrences
# sort -nr → sort by count, highest first

Other Useful Pipelines:#

# Count HTTP status codes in log
awk '{print $9}' access.log | sort | uniq -c | sort -nr

# Find most common IP addresses
awk '{print $1}' access.log | sort | uniq -c | sort -nr | head -10

# Find all POST requests
grep "POST" access.log | awk '{print $7}' | sort | uniq -c | sort -nr

# Count errors by hour
grep "ERROR" app.log | awk '{print substr($2,1,2)}' | sort | uniq -c

# Search specific pattern and count
grep -c "404" access.log

# Extract unique URLs
awk '{print $7}' access.log | sort -u

11. `dir /ah` - Windows Command (NOT Unix)#

Windows: dir /ah ← lists hidden files in Windows
Unix: ls -a ← lists hidden files in Unix/Linux/Mac

❌ dir /ah does NOT work on Unix/Linux/Mac
❌ show -a does not exist in any standard shell

12. Other Essential Shell Commands#

pwd # print working directory
cd /path/to/dir # change directory
cd .. # go up one level
cd ~ # go to home directory
cd - # go to previous directory
mkdir new_folder # create directory
mkdir -p a/b/c # create nested directories
rmdir empty_folder # remove empty directory
rm -rf folder/ # remove directory and contents (careful!)

File Operations:#

cp source.txt dest.txt # copy file
cp -r source/ dest/ # copy directory recursively
mv old.txt new.txt # move/rename file
rm file.txt # remove file
rm -f file.txt # force remove (no confirmation)
touch newfile.txt # create empty file or update timestamp

Viewing & Searching:#

less file.txt # view file page by page (better than cat for large files)
more file.txt # similar to less
nano file.txt # simple text editor
vi file.txt # powerful text editor

# Search in less:
# /search_term → forward search
# ?search_term → backward search
# n → next match
# q → quit

Process Management:#

ps aux # list all running processes
kill -9 PID # force kill process
top # real-time process monitor
htop # improved process monitor

File Permissions:#

chmod 755 script.py # set permissions (rwxr-xr-x)
chmod +x script.py # make executable
chmod 644 file.txt # rw-r--r--
chown user:group file.txt # change owner

# Permission notation:
# r=4, w=2, x=1
# 755 → rwxr-xr-x (owner: all, group+others: read+execute)
# 644 → rw-r--r-- (owner: read+write, others: read only)

Network & Downloads:#

curl https://api.example.com/data # make HTTP request
curl -O https://example.com/file.zip # download file
wget https://example.com/file.zip # download file
curl -H "Authorization: Bearer TOKEN" url # with header
curl -X POST -d '{"key":"value"}' url # POST request

Environment Variables:#

export API_KEY="your_key" # set environment variable
echo $API_KEY # print variable
env # list all environment variables
unset API_KEY # remove variable
printenv API_KEY # print specific variable

13. Shell Scripting Basics#

Shebang and Basic Script:#

#!/bin/bash
# This is a comment

# Variables
NAME="John"
echo "Hello, $NAME"

# User input
read -p "Enter name: " name
echo "Hello, $name"

# Conditionals
if [ -f "file.txt" ]; then
 echo "File exists"
elif [ -d "folder" ]; then
 echo "Folder exists"
else
 echo "Neither exists"
fi

# Loops
for i in {1..5}; do
 echo "Number: $i"
done

for file in *.csv; do
 echo "Processing: $file"
done

# While loop
while IFS= read -r line; do
 echo "$line"
done < file.txt

ETL Script with Idempotency:#

#!/bin/bash
# Idempotent ETL script

TEMP_FILE="/tmp/output_temp.csv"
FINAL_FILE="/output/final.csv"
LOG_FILE="/logs/etl.log"

echo "$(date): Starting ETL" >> $LOG_FILE

# Write to temp file first
python process_data.py > $TEMP_FILE

# Check if processing succeeded
if [ $? -eq 0 ]; then
 # Atomic move to final destination
 mv $TEMP_FILE $FINAL_FILE # ✅ atomic operation
 echo "$(date): ETL completed" >> $LOG_FILE
else
 echo "$(date): ETL failed" >> $LOG_FILE
 rm -f $TEMP_FILE
 exit 1
fi

14. `grep` for Log Analysis - Exam Scenarios#

# ✅ Correct: grep searches CONTENT
grep "Pine Ridge" evacuation-logs.txt

# ❌ Wrong: find searches FILE NAMES
find -name "Pine Ridge"

# ❌ Wrong: ls lists directory contents
ls "Pine Ridge"

# ❌ Wrong: cat shows everything, no search
cat evacuation-logs.txt

Scenario 2: Search across multiple files#

grep -r "ERROR" ./logs/ # recursive search
grep "404" *.log # all log files
grep -l "critical" ./logs/* # list files containing "critical"

Scenario 3: Extract specific log entries#

# POST requests to /checkout/
grep "POST /checkout" access.log

# Requests between 12:00-15:59
grep -E "1[2-5]:[0-5][0-9]" access.log

# Combine with awk for field extraction
grep "POST" access.log | awk '{print $1, $4, $7, $9}'

Shell Commands - Quick Reference Card#

Navigation & Files:
 pwd → current directory
 ls -a → list all files (including hidden) ✅
 ls -la → long format + hidden
 cd path → change directory
 find → search FILE NAMES

Viewing Content:
 cat → display file (small files)
 less → view file page by page (large files)
 head -n N → first N lines
 tail -n N → last N lines
 tail -f → follow file in real-time

Searching:
 grep "term" file → search CONTENT ✅
 grep -i → case-insensitive
 grep -E "pattern" → extended regex
 grep -r "term" dir → recursive search

Processing:
 awk '{print $N}' → extract field N
 sort → sort output
 sort -nr → numeric reverse sort
 uniq -c → count unique ✅
 wc -l → count lines

Pipeline:
 cmd1 | cmd2 → pipe output to next command
 grep | awk | sort | uniq -c | sort -nr ✅

Log Analysis Pattern:
 grep -E "pattern" log | awk '{print $field}' | sort | uniq -c | sort -nr

3. Python File & Memory Handling

Wed, 03 Jan 2024 00:00:00 +0000

3. Python File & Memory Handling#

1. Generator Expression - Memory-Efficient File Processing#

What is a Generator?#

Produces values one at a time on demand
Does NOT load entire data into memory
Uses yield keyword or generator expressions
Critical for processing large files (GBs)

The Problem with Loading Entire File:#

# ❌ Bad - loads ENTIRE file into memory at once
lines = open('logfile.txt').readlines() # all lines in RAM
for line in lines:
 process(line)

# If file is 10GB → needs 10GB of RAM → crash

Generator Solution:#

# ✅ Good - reads ONE line at a time
for line in open('logfile.txt'):
 process(line)

# Only one line in memory at any time
# Works for files of ANY size

Generator Expression vs List Comprehension:#

# List comprehension - loads ALL into memory ❌
results = [process(line) for line in open('file.txt')]

# Generator expression - one at a time ✅
results = (process(line) for line in open('file.txt'))

# Difference:
# List: [...] → eager, all in memory
# Generator: (...) → lazy, one at a time

Generator with `yield`:#

def process_logs(filepath):
 """Generator function - yields one line at a time"""
 with open(filepath, 'r') as f:
 for line in f:
 line = line.strip()
 if line: # skip empty lines
 yield parse_line(line)

# Usage - memory efficient
for entry in process_logs('access.log'):
 if entry['status'] == 404:
 print(entry['url'])

# vs loading all at once ❌
all_entries = list(process_logs('access.log')) # defeats purpose

Practical Example - Large Log Processing:#

def process_large_log(filepath, filter_func):
 """
 Process large log file without memory issues
 ✅ Generator pattern - exam answer for large file processing
 """
 with open(filepath, 'r') as f:
 for line in f: # reads ONE line at a time
 entry = parse_line(line.strip())
 if entry and filter_func(entry):
 yield entry

# Count POST requests - never loads full file
post_count = sum(
 1 for entry in process_large_log('access.log',
 lambda e: e['method'] == 'POST')
)

# Memory usage: constant regardless of file size ✅
# vs readlines(): proportional to file size ❌

2. `readlines()` - Why Avoid for Large Files#

# ❌ readlines() - loads entire file into list
lines = open('logfile').readlines()
# Returns: ['line1\n', 'line2\n', 'line3\n', ...]
# Problem: 10GB file = 10GB in RAM

# ❌ read() - even worse, single giant string
content = open('logfile').read()

# ❌ pandas for log files - loads everything
df = pd.read_csv('logfile.csv') # all rows in RAM

# ✅ Correct alternatives:
# 1. Generator (for line in open())
# 2. pd.read_csv(chunksize=N)
# 3. gzip.open() for compressed files

Memory Comparison:#

File Size readlines() Generator
─────────────────────────────────────
1 MB 1 MB RAM ~few KB
100 MB 100 MB RAM ~few KB
1 GB 1 GB RAM ~few KB
10 GB CRASH ~few KB

3. `pd.read_csv(chunksize=N)` - Large CSV Processing#

Basic Usage:#

import pandas as pd

# ✅ Process large CSV in chunks
for chunk in pd.read_csv('large_file.csv', chunksize=10000):
 # chunk is a regular DataFrame with 10000 rows
 process(chunk)

# Each chunk is processed and discarded before next is loaded
# Memory: only 10000 rows at a time regardless of file size

Complete Chunked Processing:#

import pandas as pd

def process_large_csv(filepath, chunksize=10000):
 """Process large CSV without memory issues"""
 results = []

 for i, chunk in enumerate(pd.read_csv(filepath, chunksize=chunksize)):
 print(f"Processing chunk {i+1}...")

 # Apply operations to chunk
 chunk['normalized'] = chunk['name'].str.lower()
 filtered = chunk[chunk['sales'] > 1000]
 results.append(filtered)

 # Combine results
 final = pd.concat(results, ignore_index=True)
 return final

# Usage
df = process_large_csv('sales_data.csv')

Aggregation Across Chunks:#

import pandas as pd

# ✅ Calculate statistics without loading all data
total_sales = 0
count = 0

for chunk in pd.read_csv('sales.csv', chunksize=10000):
 total_sales += chunk['sales'].sum()
 count += len(chunk)

average_sales = total_sales / count
print(f"Average sales: {average_sales:.2f}")

Chunked Processing with Filter:#

# Find all rows matching condition in large file
matching_rows = []

for chunk in pd.read_csv('data.csv', chunksize=10000):
 # Filter within chunk
 filtered = chunk[
 (chunk['status'] == 'active') &
 (chunk['amount'] > 1000)
 ]
 matching_rows.append(filtered)

result = pd.concat(matching_rows, ignore_index=True)

Options with `read_csv`:#

# Useful options for large files
pd.read_csv(
 'large.csv',
 chunksize=10000, # rows per chunk ✅
 usecols=['a', 'b'], # load only needed columns → saves memory
 dtype={'id': str}, # specify dtypes → prevents auto-conversion
 nrows=1000, # load only first N rows (for testing)
 skiprows=range(1, 100), # skip rows
 encoding='utf-8', # specify encoding
 low_memory=False # better dtype inference
)

4. `gzip.open()` - Reading Compressed Files#

Why Gzipped Files?#

Log files are often compressed to save disk space
.gz extension = gzip compressed
Python can read them directly without manual decompression

Basic Usage:#

import gzip

# ✅ Read gzipped text file
with gzip.open('logfile.gz', 'rt') as f: # 'rt' = read text
 for line in f: # generator pattern
 process(line)

# Read entire compressed file (small files only)
with gzip.open('logfile.gz', 'rt') as f:
 content = f.read()

# Write gzipped file
with gzip.open('output.gz', 'wt') as f:
 f.write("compressed content")

Mode Options:#

gzip.open('file.gz', 'rt') # read text ✅ most common
gzip.open('file.gz', 'rb') # read binary
gzip.open('file.gz', 'wt') # write text
gzip.open('file.gz', 'wb') # write binary
gzip.open('file.gz', 'at') # append text

Complete Log Analysis - Gzipped Apache Log:#

import gzip
from datetime import datetime

def analyze_gzipped_log(filepath):
 """
 Analyze gzipped Apache log file
 Uses generator pattern for memory efficiency
 """
 post_count = 0
 status_counts = {}

 with gzip.open(filepath, 'rt') as f:
 for line in f: # generator - one line at a time
 entry = parse_apache_log(line.strip())
 if entry is None:
 continue

 # Count POST requests
 if entry['method'] == 'POST':
 post_count += 1

 # Count by status code
 status = entry['status']
 status_counts[status] = status_counts.get(status, 0) + 1

 return post_count, status_counts

# Run analysis
posts, statuses = analyze_gzipped_log('access.log.gz')
print(f"POST requests: {posts}")
print(f"Status distribution: {statuses}")

pandas with Gzipped CSV:#

# pandas auto-detects .gz extension
df = pd.read_csv('data.csv.gz') # small files
for chunk in pd.read_csv('data.csv.gz', chunksize=10000): # large files
 process(chunk)

5. `with open()` - Context Manager#

Why Use Context Manager?#

Automatically closes file when done
Works even if exception occurs
Prevents resource leaks

# ✅ Always use context manager
with open('file.txt', 'r') as f:
 content = f.read()
# File automatically closed here, even if error occurs

# ❌ Bad - manual close, risky
f = open('file.txt', 'r')
content = f.read()
f.close() # might not run if error occurs above

File Modes:#

open('file.txt', 'r') # read text (default)
open('file.txt', 'w') # write text (creates/overwrites)
open('file.txt', 'a') # append text
open('file.txt', 'x') # exclusive creation (fails if exists)
open('file.txt', 'rb') # read binary
open('file.txt', 'wb') # write binary
open('file.txt', 'r+') # read and write

Reading Options:#

with open('file.txt', 'r') as f:
 # Read entire file
 content = f.read() # single string

 # Read all lines into list
 lines = f.readlines() # ['line1\n', 'line2\n']

 # Read one line
 line = f.readline() # 'line1\n'

 # ✅ Iterate line by line (memory efficient)
 for line in f:
 process(line.strip()) # strip removes \n

Writing Options:#

with open('output.txt', 'w') as f:
 f.write("Hello World\n") # write string
 f.writelines(["line1\n", "line2\n"]) # write list

# Append to existing file
with open('output.txt', 'a') as f:
 f.write("New line\n")

Encoding:#

# Always specify encoding for non-ASCII content
with open('file.txt', 'r', encoding='utf-8') as f:
 content = f.read()

with open('file.txt', 'w', encoding='utf-8') as f:
 f.write("Content with unicode: é, ü, 中文")

6. `os.path` - File Path Operations#

import os

# Check existence
os.path.exists('file.txt') # True if exists (file or dir)
os.path.isfile('file.txt') # True if file
os.path.isdir('folder/') # True if directory

# Path manipulation
os.path.join('folder', 'file.txt') # 'folder/file.txt' (OS-safe)
os.path.basename('/path/to/file.txt') # 'file.txt'
os.path.dirname('/path/to/file.txt') # '/path/to'
os.path.splitext('file.txt') # ('file', '.txt')
os.path.abspath('file.txt') # absolute path

# File info
os.path.getsize('file.txt') # size in bytes
os.path.getmtime('file.txt') # last modified time

# Directory operations
os.listdir('.') # list directory
os.makedirs('path/to/dir', exist_ok=True) # create dirs
os.remove('file.txt') # delete file
os.rename('old.txt', 'new.txt') # rename file

# Environment
os.getcwd() # current working directory
os.chdir('/path/to/dir') # change directory
os.environ.get('API_KEY') # get environment variable
os.getenv('API_KEY', 'default') # with default value

7. `pathlib.Path` - Modern File Path Handling#

Why pathlib?#

More readable and Pythonic than os.path
Works consistently across Windows/Mac/Linux
Object-oriented approach

from pathlib import Path

# Create path object
p = Path('data/file.txt')
p = Path('/absolute/path/file.txt')

# Check existence
p.exists() # True/False
p.is_file() # True if file
p.is_dir() # True if directory

# Path components
p.name # 'file.txt'
p.stem # 'file'
p.suffix # '.txt'
p.parent # Path('data')
p.parts # ('data', 'file.txt')

# Build paths (OS-safe)
base = Path('data')
full = base / 'subfolder' / 'file.txt' # ✅ clean syntax

# File operations
p.read_text() # read entire file as string
p.write_text("content") # write string to file
p.read_bytes() # read as bytes
p.write_bytes(data) # write bytes

# Create directories
Path('new/nested/dir').mkdir(parents=True, exist_ok=True)

# List directory
for file in Path('data').iterdir():
 print(file)

# Find files by pattern
for csv_file in Path('.').glob('**/*.csv'): # recursive
 print(csv_file)

for log_file in Path('logs').glob('*.log'): # non-recursive
 print(log_file)

# Delete
p.unlink() # delete file
p.rmdir() # delete empty directory

# Rename/move
p.rename('new_name.txt')
p.replace('new_location/file.txt')

8. `tempfile` - Temporary Files#

Why Temporary Files?#

Safe place to write data during processing
Automatically cleaned up
Essential for idempotent ETL pipelines

import tempfile
import os
import shutil

# Create temporary file
with tempfile.NamedTemporaryFile(
 mode='w',
 suffix='.csv',
 delete=False # don't delete on close
) as tmp:
 tmp.write("data,goes,here\n")
 tmp_path = tmp.name # save path for later

# File persists after 'with' block (because delete=False)
print(f"Temp file at: {tmp_path}")

# ✅ Idempotent ETL pattern:
def safe_write_output(data, final_path):
 """Write to temp first, then atomic move to final"""

 # Step 1: Write to temp file
 tmp = tempfile.NamedTemporaryFile(
 mode='w',
 suffix='.csv',
 delete=False
 )
 try:
 for row in data:
 tmp.write(row + '\n')
 tmp.close()

 # Step 2: Atomic move to final destination
 shutil.move(tmp.name, final_path) # ✅ atomic
 print(f"Output written to: {final_path}")

 except Exception as e:
 # Cleanup temp file on error
 tmp.close()
 os.unlink(tmp.name)
 raise e

# Create temp directory
with tempfile.TemporaryDirectory() as tmpdir:
 # Work inside temp directory
 work_file = os.path.join(tmpdir, 'work.csv')
 # Automatically deleted when 'with' block exits

9. `shutil` - File Operations#

import shutil

# Copy file
shutil.copy('source.txt', 'dest.txt') # copy file
shutil.copy2('source.txt', 'dest.txt') # copy with metadata
shutil.copytree('source_dir/', 'dest_dir/') # copy directory

# Move file (ATOMIC on same filesystem) ✅
shutil.move('temp_output.csv', 'final_output.csv')

# Delete
shutil.rmtree('directory/') # delete directory and contents

# Archive
shutil.make_archive('output', 'zip', 'folder/') # create zip
shutil.unpack_archive('archive.zip', 'dest/') # extract

# Disk usage
shutil.disk_usage('/') # total, used, free bytes

Why `shutil.move()` is Atomic:#

# ✅ Atomic move - all or nothing
shutil.move('temp.csv', 'final.csv')
# If crash occurs DURING move:
# → temp.csv may be deleted
# → final.csv still has OLD content
# → No partial/corrupt file

# ❌ Non-atomic write
with open('final.csv', 'w') as f:
 for row in data:
 f.write(row) # crash here = partial/corrupt file

Memory Management - Complete Reference#

Python Memory Concepts:#

import sys

# Check object size in bytes
sys.getsizeof([1, 2, 3]) # size of list
sys.getsizeof("hello") # size of string

# Memory-efficient alternatives:
# 1. Generator vs List
gen = (x**2 for x in range(1000000)) # tiny memory
lst = [x**2 for x in range(1000000)] # large memory

# 2. numpy arrays vs Python lists
import numpy as np
arr = np.array([1, 2, 3], dtype=np.int32) # 4 bytes per element
lst = [1, 2, 3] # ~28 bytes per element

# 3. Chunked processing vs loading all
# (covered in pd.read_csv(chunksize=N))

# 4. Delete large objects when done
large_data = load_large_dataset()
process(large_data)
del large_data # free memory
import gc
gc.collect() # force garbage collection

Quick Reference - When to Use What#

File Size Approach
───────────────────────────────────────────────
Small (<100MB) → pd.read_csv() or open().read()
Medium (100MB-1GB) → pd.read_csv(chunksize=N)
Large (>1GB) → Generator: for line in open()
Compressed (.gz) → gzip.open('file.gz', 'rt')
CSV compressed → pd.read_csv('file.csv.gz')

File Operation → Method
───────────────────────────────────────────────
Read text file → with open('f', 'r') as f
Write safely → tempfile + shutil.move() ✅
Check existence → os.path.exists() or Path.exists()
Build paths → os.path.join() or Path / 'file'
Find files → Path.glob('**/*.csv')
Process large → Generator expression ✅
Chunk large CSV → pd.read_csv(chunksize=N) ✅

4. Git & Version Control

Thu, 04 Jan 2024 00:00:00 +0000

4. Git & Version Control#

1. Git - Core Concepts & Purpose#

What is Git?#

Distributed version control system
Tracks changes to files over time
Every developer has full copy of repository
Works offline - syncs when connected

Core Benefits:#

✅ Tracks every change ever made
✅ Enables collaboration among multiple developers
✅ Rollback to any previous state
✅ Branching - work on features without affecting main code
✅ Blame - see who changed what and when
✅ Merge - combine work from multiple people

❌ Does NOT automatically clean data
❌ Does NOT replace testing
❌ Does NOT store only binary assets

Git vs GitHub vs GitLab:#

Tool	What it is
Git	Version control software (local)
GitHub	Cloud hosting for Git repos + collaboration
GitLab	Alternative to GitHub (self-hostable)
Bitbucket	Another alternative by Atlassian

2. Git - Core Architecture#

Working Directory Staging Area Local Repo Remote Repo
(your files) (index) (.git folder) (GitHub)
 │ │ │ │
 │ git add │ git commit │ git push │
 │──────────────────>│────────────────>│───────────────>│
 │ │ │ │
 │<──────────────────────────────────────────────────── │
 │ git pull / git clone │
 │ │ │ │
 │ git checkout -- . git fetch │
 │<──────────────────│ │<───────────────│

Three Areas Explained:#

# 1. Working Directory
# → Files you're currently editing
# → Changes not yet tracked by Git

# 2. Staging Area (Index)
# → Files marked for next commit
# → git add moves files here

# 3. Local Repository
# → Committed snapshots
# → git commit moves from staging to here

# 4. Remote Repository
# → GitHub/GitLab server
# → git push uploads local commits
# → git pull downloads remote commits

3. `.gitignore` - Excluding Files from Tracking#

What is `.gitignore`?#

Text file listing patterns of files Git should NOT track
Placed in root of repository
Committed to repo so all team members use same rules

Common `.gitignore` Entries:#

# Python
__pycache__/
*.pyc
*.pyo
*.pyd
.Python
*.egg-info/

# Virtual environments
venv/
.venv/
env/
.env

# Secrets & API keys ✅ most important
.env
.env.*
secrets.json
credentials.json
config/secrets.py

# Data files (often too large for git)
data/raw/
*.csv
*.parquet
*.pkl

# IDE files
.vscode/
.idea/
*.swp

# OS files
.DS_Store # Mac
Thumbs.db # Windows

# Jupyter
.ipynb_checkpoints/

# Build artifacts
dist/
build/
*.egg-info/

# Logs
*.log
logs/

`.gitignore` vs `.git/config`:#

File	Purpose
`.gitignore`	Specify files Git should NOT track ✅
`.git/config`	Repository settings (remote URL, branches)
`.gitattributes`	File-specific settings (line endings, merge strategy)

❌ NOT for tracking specific files
❌ NOT for listing contributors
❌ NOT for defining repository settings

4. `git diff` - View Changes#

# Working directory vs last commit
git diff # ✅ exam answer

# Staged changes vs last commit
git diff --staged
git diff --cached # same as --staged

# Between two commits
git diff abc123 def456

# Between two branches
git diff main feature-branch

# Specific file
git diff file.py

# Summary only (no actual diff)
git diff --stat

git status # shows WHICH files changed (not the actual diff)
git log # shows commit HISTORY
git show # shows changes in a specific commit
git log -p # shows commits WITH their diffs

`git diff` Output:#

diff --git a/script.py b/script.py
index abc123..def456 100644
--- a/script.py ← old version
+++ b/script.py ← new version
@@ -10,7 +10,8 @@ ← line numbers
 def clean_data(df):
- df.dropna() ← removed line (red)
+ df.dropna(subset=['name']) ← added line (green)
+ df.reset_index(drop=True) ← added line (green)
 return df

5. Git Workflow Commands - Complete Reference#

Staging & Committing:#

# Check what changed
git status # see modified/staged/untracked files

# Stage changes
git add file.py # stage specific file
git add . # stage ALL changes ✅
git add *.py # stage all Python files
git add -p # stage changes interactively (chunk by chunk)

# Unstage (undo git add)
git restore --staged file.py
git reset HEAD file.py # older syntax

# Commit
git commit -m "Add data cleaning function" # ✅
git commit -am "message" # stage + commit tracked files in one step
git commit --amend # modify last commit message

# Correct sequence for uploading work:
git add .
git commit -m "message"
git push # ✅ exam answer: add → commit → push

Viewing History:#

git log # full commit history
git log --oneline # compact one-line per commit
git log --graph # visual branch graph
git log --oneline --graph --all # full visual history
git log -n 5 # last 5 commits
git log --author="John" # commits by specific author
git log --since="2024-01-01" # commits after date
git log -- file.py # commits touching specific file
git show abc123 # show specific commit details
git show HEAD # show last commit
git show HEAD~1 # show second-to-last commit

6. Syncing with Remote#

# Download + integrate remote changes
git pull # pull from tracked remote branch
git pull origin main # ✅ pull from main branch specifically

# Download WITHOUT integrating
git fetch origin # fetch all remote changes
git fetch origin main # fetch specific branch

# Pull vs Fetch:
# git pull = git fetch + git merge
# git fetch = download only (safe, no changes to working files)

# Push local commits to remote
git push # push to tracked remote branch
git push origin main # push to specific branch
git push origin feature # push feature branch
git push -u origin feature # push + set upstream tracking
git push --force # ⚠️ force push (dangerous, avoid)

7. Git Branching#

Why Branches?#

Work on features without affecting main code
Multiple people work simultaneously without conflicts
Keep main always production-ready

# Create branch
git branch feature-analysis # create branch
git branch # list local branches
git branch -a # list all branches (local + remote)
git branch -d feature-analysis # delete branch (safe)
git branch -D feature-analysis # force delete branch

# Switch branch
git checkout feature-analysis # switch to branch
git checkout main # go back to main

# Create + switch in one step ✅
git checkout -b feature-analysis # ✅ most common shortcut
git switch -c feature-analysis # newer syntax (Git 2.23+)

# ✅ Exam answer: create and switch
git branch feature-analysis
git checkout feature-analysis
# OR single command:
git checkout -b feature-analysis

Branch Naming Conventions:#

feature/data-cleaning # new features
bugfix/fix-null-handling # bug fixes
hotfix/critical-api-fix # urgent production fixes
release/v1.2.0 # release preparation
docs/update-readme # documentation

8. Undoing Changes#

# Discard working directory changes ✅ (exam answer)
git checkout -- . # discard ALL unstaged changes
git checkout -- file.py # discard changes to specific file
git restore . # newer syntax (Git 2.23+)
git restore file.py # specific file

# Unstage (undo git add)
git restore --staged file.py
git reset HEAD file.py

# Undo last commit (keep changes in working directory)
git reset --soft HEAD~1 # undo commit, keep staged
git reset HEAD~1 # undo commit, unstage changes
git reset --mixed HEAD~1 # same as above (default)

# Undo last commit (DISCARD all changes) ⚠️ destructive
git reset --hard HEAD~1

# Revert (creates NEW commit that undoes previous)
git revert HEAD # undo last commit safely
git revert abc123 # undo specific commit

# Stash (save work temporarily)
git stash # save current changes
git stash list # see all stashes
git stash pop # apply most recent stash + delete it
git stash apply # apply most recent stash (keep it)
git stash drop # delete most recent stash
git stash clear # delete all stashes

Undo Command Comparison:#

Command	Effect	Destructive?
`git checkout -- .`	Discard working dir changes ✅	Yes (local only)
`git reset --soft`	Undo commit, keep staged	No
`git reset --hard`	Undo commit, discard all	Yes ⚠️
`git revert`	New commit undoing previous	No (safe)
`git stash`	Temporarily save changes	No
`git clean -fd`	Remove untracked files	Yes ⚠️

9. Merging & Merge Conflicts#

Merging Branches:#

# Merge feature into main
git checkout main # switch to target branch
git merge feature-analysis # merge feature into current branch

# Fast-forward merge (no new commit needed)
git merge --ff feature

# Always create merge commit
git merge --no-ff feature

# Merge with message
git merge -m "Merge feature-analysis" feature

Merge Conflicts:#

# When do conflicts occur?
# → Two people edit the SAME LINE on different branches
# → Git cannot auto-decide which change to keep
# → Must be RESOLVED MANUALLY ✅ (exam answer)

# After conflict:
git status # shows conflicted files

# Conflicted file looks like:
<<<<<<< HEAD (current branch)
df.dropna(subset=['name']) ← your change
=======
df.dropna(subset=['email']) ← incoming change
>>>>>>> feature-branch

# Resolution steps:
# 1. Open conflicted file
# 2. Choose which change to keep (or combine both)
# 3. Remove conflict markers (<<<<, ====, >>>>)
# 4. Stage resolved file
git add file.py
# 5. Complete merge
git commit

# Abort merge (go back to before merge)
git merge --abort

10. Pull Requests (PR)#

What is a PR?#

Formal request to merge your branch into main
Team members can:
- Review code changes
- Leave comments
- Request changes
- Approve the merge
Standard practice in collaborative projects

PR Workflow:#

1. Create feature branch
 git checkout -b feature-data-cleaning

2. Make changes and commit
 git add .
 git commit -m "Add data cleaning pipeline"

3. Push branch to remote
 git push origin feature-data-cleaning

4. Create PR on GitHub/GitLab
 → Go to repository on GitHub
 → Click "New Pull Request"
 → Select base: main, compare: feature-data-cleaning
 → Add title and description
 → Request reviewers

5. Code review
 → Reviewers comment on specific lines
 → Author makes requested changes
 → Push additional commits to same branch

6. Approval and merge
 → Reviewer approves
 → PR merged into main

7. Clean up
 git branch -d feature-data-cleaning
 git push origin --delete feature-data-cleaning

✅ PRs allow code review + approval before merging
❌ PRs are NOT faster than direct merging
❌ PRs do NOT automatically fix bugs
❌ PRs are NOT only for documentation

11. Git Hotfix Workflow#

When to Use Hotfix?#

Critical bug found in production
Cannot wait for regular release cycle
Need immediate fix with minimal disruption

Hotfix Process:#

# Step 1: Create hotfix branch from LAST RELEASE TAG
git checkout main
git checkout -b hotfix/critical-routing-bug

# Step 2: Implement and test the fix
# (make changes, run tests)
git add .
git commit -m "Fix critical routing bug in intersection algorithm"

# Step 3: Open PR for expedited review
git push origin hotfix/critical-routing-bug
# → Create PR on GitHub
# → Request urgent/expedited review

# Step 4: Merge into BOTH main AND develop ✅ (exam answer)
git checkout main
git merge hotfix/critical-routing-bug
git tag -a v1.0.1 -m "Hotfix: critical routing bug"
git push origin main --tags

git checkout develop
git merge hotfix/critical-routing-bug
git push origin develop

# Step 5: Delete hotfix branch
git branch -d hotfix/critical-routing-bug
git push origin --delete hotfix/critical-routing-bug

✅ Create hotfix branch from release tag → fix → PR → merge to main AND develop
❌ Apply fix directly to main without review
❌ Delay fix to next regular release cycle
❌ Fix in working directory, apply to production, backport to Git later

12. Branch Strategy - GitFlow#

Complete Branching Model:#

main (production-ready)
 │
 ├── hotfix branches (emergency fixes)
 │ └── merge back to main + develop
 │
develop (integration branch)
 │
 ├── feature/data-cleaning
 ├── feature/visualization
 ├── feature/api-integration
 │ └── merge back to develop when complete
 │
 └── release/v1.2.0 (when ready for production)
 └── merge to main + develop, then tag

Simple GitHub Flow (simpler alternative):#

main (always deployable)
 │
 ├── feature/data-cleaning → PR → main
 ├── bugfix/fix-null-values → PR → main
 └── hotfix/urgent-fix → PR → main (expedited)

13. Pipeline Transformation Versioning#

Problem:#

Transformation logic changes frequently
→ Hard to track what changed and when
→ Can't reproduce old results
→ Risk of breaking existing processes

Solution:#

# ✅ Version transformations with git + unit tests
git checkout -b feature/new-transformation-logic

# Write unit tests FIRST
# tests/test_transform.py

# Implement transformation
# src/transform.py

# Test on sample data (not production)
pytest tests/ -v

# If all tests pass → PR → merge
git add .
git commit -m "Update normalization: handle unicode names"
git push origin feature/new-transformation-logic

# Unit test for transformation
def test_normalize_company_name():
 assert normalize("APPLE INC") == "Apple Inc"
 assert normalize("apple incorporated") == "Apple Inc"
 assert normalize(" Apple ") == "Apple"

def test_schema_preserved():
 result = transform(sample_df)
 assert list(result.columns) == ['id', 'name', 'revenue']
 assert result['revenue'].dtype == 'float64'

✅ Version with git + write unit tests + test on sample data before production
❌ Keep all logic in one script and update directly
❌ Manually document changes in shared doc
❌ Avoid changes to prevent breaking things

Git Commands - Complete Quick Reference#

# SETUP
git config --global user.name "Name"
git config --global user.email "email"
git init # initialize new repo
git clone url # clone existing repo

# DAILY WORKFLOW
git status # what changed?
git diff # what exactly changed? ✅
git add . # stage all ✅
git commit -m "message" # save snapshot ✅
git push # upload to remote ✅
git pull origin main # get latest ✅

# BRANCHING
git branch # list branches
git checkout -b feature # create + switch ✅
git checkout main # switch branch
git merge feature # merge branch
git branch -d feature # delete branch

# UNDOING
git checkout -- . # discard changes ✅
git restore --staged file # unstage
git reset --soft HEAD~1 # undo last commit (keep changes)
git reset --hard HEAD~1 # undo last commit (discard) ⚠️
git revert HEAD # safe undo (new commit)
git stash / git stash pop # save/restore temp changes

# REMOTE
git remote -v # show remote URLs
git fetch origin # download (don't merge)
git pull origin main # download + merge ✅
git push origin feature # push branch

# INSPECTION
git log --oneline --graph # visual history
git show abc123 # show commit details
git blame file.py # who changed each line
git tag -a v1.0 -m "msg" # create tag

# HOTFIX
git checkout -b hotfix/fix # from main ✅
# → fix → commit → PR
# → merge to main AND develop ✅

Git - Exam Scenario Answers#

Scenario	Command
Get latest team updates	`git pull` ✅
See what changed in files	`git diff` ✅
Save work and upload	`git add . → git commit → git push` ✅
Create and switch branch	`git checkout -b feature-name` ✅
Discard working directory changes	`git checkout -- .` ✅
Files Git should not track	`.gitignore` ✅
Collaborative review before merge	Pull Request (PR) ✅
Critical production bug fix	Hotfix branch → merge to main + develop ✅
Two people edit same line	Merge conflict → manual resolution ✅

5. HTTP & APIs

Fri, 05 Jan 2024 00:00:00 +0000

5. HTTP & APIs#

1. HTTP Status Codes - Complete Reference#

What are HTTP Status Codes?#

3-digit numbers sent by server in response to client request
Indicate what happened with the request
First digit indicates category

1xx - Informational#

Server received request, processing continues

100 Continue → Client should continue sending request body
101 Switching → Server agrees to switch protocols
 Protocols (e.g., HTTP → WebSocket)
102 Processing → Server processing, no response yet

2xx - Success ✅#

Request successfully received, understood, and accepted

200 OK → Standard success ✅ most common
201 Created → Resource created (after POST) ✅
202 Accepted → Request accepted but not yet processed
204 No Content → Success but no response body (DELETE) ✅
206 Partial Content → Partial data returned (file download resume)

# Checking success in Python:
200 <= status < 300 # ✅ correct range check (exam answer)
status == 200 # ❌ misses 201, 204 etc.
status in [200, 201] # ❌ incomplete
status < 400 # ❌ includes redirects

3xx - Redirection#

Further action needed to complete request

300 Multiple Choices → Multiple options for resource
301 Moved Permanently → Resource permanently at new URL
302 Found → Resource temporarily at different URL
303 See Other → Redirect to another URL using GET
304 Not Modified → Cached version still valid, no new data ✅
307 Temp Redirect → Like 302 but must use same HTTP method
308 Perm Redirect → Like 301 but must use same HTTP method

# Checking redirect in Python:
300 <= status < 400 # ✅ correct range check (exam answer)
status in [301, 302] # ❌ incomplete
200 < status < 400 # ❌ includes some 2xx codes
status in range(299, 400) # ❌ starts at 299

4xx - Client Errors#

Request contains bad syntax or cannot be fulfilled

400 Bad Request → Malformed syntax or invalid parameters
401 Unauthorized → Authentication required/failed ✅ exam
 (missing or invalid API key)
403 Forbidden → Authenticated but no permission
404 Not Found → Resource doesn't exist at URL
405 Method Not Allowed→ HTTP method not supported for endpoint
408 Request Timeout → Server timed out waiting for request
409 Conflict → Conflicts with current state (duplicate)
410 Gone → Resource permanently deleted
413 Payload Too Large → Request body exceeds server limit
415 Unsupported → Server doesn't support content type
 Media Type
422 Unprocessable → Syntactically correct but semantically wrong
 Entity
429 Too Many Requests → Rate limit exceeded ✅ exam

5xx - Server Errors#

Server failed to fulfill valid request

500 Internal Server → Generic server-side error ✅
 Error
501 Not Implemented → Server doesn't support functionality
502 Bad Gateway → Invalid response from upstream server
503 Service → Server temporarily down
 Unavailable (overloaded or maintenance)
504 Gateway Timeout → Upstream server didn't respond in time
505 HTTP Version → Server doesn't support HTTP version
 Not Supported

Key Distinctions - Exam Focus:#

401 vs 403:
→ 401 = not authenticated yet (no/wrong credentials)
→ 403 = authenticated but no permission

401 vs 429:
→ 401 = wrong/missing API key
→ 429 = valid key but too many requests

404 vs 410:
→ 404 = not found (may exist elsewhere)
→ 410 = permanently deleted

301 vs 302:
→ 301 = permanent redirect (update your bookmarks)
→ 302 = temporary redirect (keep old URL)

500 vs 503:
→ 500 = code error on server
→ 503 = server temporarily unavailable

502 vs 504:
→ 502 = bad response FROM upstream
→ 504 = upstream TIMED OUT

2. HTTP Methods - Complete Reference#

All HTTP Methods:#

GET → Retrieve data (safe + idempotent)
POST → Send/create data (not safe, not idempotent)
PUT → Update/replace resource (idempotent)
PATCH → Partial update (not idempotent)
DELETE → Remove resource (idempotent)
HEAD → Like GET but no body (just headers)
OPTIONS → Get allowed methods for resource

Safe vs Idempotent:#

Safe = does NOT modify server state
Idempotent = same result if called multiple times

 Safe Idempotent
GET ✅ ✅
HEAD ✅ ✅
OPTIONS ✅ ✅
POST ❌ ❌
PUT ❌ ✅
PATCH ❌ ❌
DELETE ❌ ✅

✅ GET = safe (does not modify server state) - exam answer
✅ PUT = idempotent (multiple identical requests = same effect) - exam answer
❌ POST = NOT idempotent (creates new resource each call)

GET - Retrieve Data:#

import requests

# Basic GET request
response = requests.get('https://api.openweathermap.org/data/2.5/weather',
 params={
 'q': 'Delhi',
 'appid': os.getenv('API_KEY')
 }
)

# Check status
print(response.status_code) # 200
print(response.json()) # parsed JSON
print(response.text) # raw text
print(response.headers) # response headers

POST - Send Data / LLM Inference:#

# POST to LLM API (exam-relevant)
response = requests.post(
 'https://api.anthropic.com/v1/messages',
 headers={
 'x-api-key': os.getenv('ANTHROPIC_API_KEY'),
 'Content-Type': 'application/json'
 },
 json={
 'model': 'claude-3-sonnet-20240229',
 'max_tokens': 100,
 'messages': [
 {'role': 'user', 'content': 'Classify sentiment: Great product!'}
 ]
 }
)

data = response.json()
print(data['content'][0]['text'])

PUT - Update Resource:#

# PUT replaces entire resource
response = requests.put(
 'https://api.example.com/users/123',
 json={'name': 'John', 'email': 'john@example.com'}
)
# Calling same PUT 5 times = same result as calling once (idempotent)

PATCH - Partial Update:#

# PATCH updates only specified fields
response = requests.patch(
 'https://api.example.com/users/123',
 json={'email': 'newemail@example.com'} # only update email
)

DELETE:#

response = requests.delete('https://api.example.com/users/123')
print(response.status_code) # 204 No Content (success)

3. API Rate Limiting - Handling#

What is Rate Limiting?#

APIs limit number of requests per time period
Example: 60 calls/minute (OpenWeatherMap free tier)
Exceeding limit → HTTP 429 Too Many Requests

Strategies:#

1. Simple Sleep Between Requests:#

import requests
import time

cities = ['Delhi', 'Mumbai', 'Chennai', 'Kolkata', 'Bangalore']
results = []

for city in cities:
 response = requests.get(
 'https://api.openweathermap.org/data/2.5/weather',
 params={'q': city, 'appid': os.getenv('API_KEY')}
 )

 if response.status_code == 200:
 results.append(response.json())
 elif response.status_code == 429:
 print("Rate limit hit, waiting...")
 time.sleep(60) # wait 1 minute
 # Retry
 response = requests.get(...)
 results.append(response.json())

 time.sleep(1) # ✅ 1-2 sec between requests (exam answer)

print(f"Collected data for {len(results)} cities")

2. Exponential Backoff:#

import time
import requests

def fetch_with_retry(url, params, max_retries=3):
 """Retry with exponential backoff on rate limit"""
 for attempt in range(max_retries):
 response = requests.get(url, params=params)

 if response.status_code == 200:
 return response.json()

 elif response.status_code == 429:
 wait_time = 2 ** attempt # 1, 2, 4 seconds
 print(f"Rate limited. Waiting {wait_time}s...")
 time.sleep(wait_time)

 elif response.status_code >= 500:
 wait_time = 2 ** attempt
 print(f"Server error {response.status_code}. Retrying...")
 time.sleep(wait_time)

 else:
 raise Exception(f"Unexpected status: {response.status_code}")

 raise Exception("Max retries exceeded")

3. Using `tenacity` Library:#

from tenacity import (
 retry,
 stop_after_attempt,
 wait_exponential,
 retry_if_exception_type
)

@retry(
 stop=stop_after_attempt(3),
 wait=wait_exponential(multiplier=1, min=1, max=60)
)
def fetch_weather(city):
 response = requests.get(url, params={'q': city})
 if response.status_code == 429:
 raise Exception("Rate limited")
 return response.json()

✅ Use time.sleep(1-2) between requests - exam answer
❌ Make all requests simultaneously → triggers 429
❌ Ignore rate limit → API blocks your key
❌ Request only during off-peak hours → not reliable

4. API Security - Storing Keys#

Environment Variables - Best Practice:#

import os

# ✅ Read from environment variable
api_key = os.getenv('WEATHER_API_KEY')
api_key = os.environ.get('WEATHER_API_KEY')
api_key = os.getenv('WEATHER_API_KEY', 'default_value') # with default

# Usage
response = requests.get(url, params={
 'q': 'Delhi',
 'appid': api_key # ✅ from environment
})

Setting Environment Variables:#

# Linux/Mac - terminal
export WEATHER_API_KEY="your_key_here"
export ANTHROPIC_API_KEY="sk-ant-..."

# Windows - command prompt
set WEATHER_API_KEY=your_key_here

# Windows - PowerShell
$env:WEATHER_API_KEY = "your_key_here"

# Persistent - add to ~/.bashrc or ~/.zshrc
echo 'export WEATHER_API_KEY="your_key"' >> ~/.bashrc
source ~/.bashrc

Using `.env` File:#

# .env file (NEVER commit to git!)
WEATHER_API_KEY=your_key_here
ANTHROPIC_API_KEY=sk-ant-...
DATABASE_URL=postgresql://user:pass@localhost/db
DEBUG=false

# Load .env file with python-dotenv
from dotenv import load_dotenv
import os

load_dotenv() # loads .env file
api_key = os.getenv('WEATHER_API_KEY') # read variable

# .gitignore - ALWAYS exclude .env
.env
.env.*
*.env
secrets.json

What NOT to Do:#

# ❌ Hard-coded in script (visible in git history)
api_key = "abc123xyz789"

# ❌ In code comments
# API key: abc123xyz789

# ❌ In GitHub README
# Use API key: abc123xyz789

# ❌ Printed to console (appears in logs)
print(f"Using key: {api_key}")

5. JSON Response Parsing - Nested Fields#

Understanding Nested JSON:#

# Typical API response structure
response_json = {
 "name": "Delhi",
 "main": {
 "temp": 298.15,
 "humidity": 60
 },
 "weather": [ # ← this is a LIST
 {
 "description": "clear sky",
 "icon": "01d"
 }
 ],
 "wind": {
 "speed": 3.5
 }
}

# ✅ Correct access patterns:
city_name = response_json['name'] # "Delhi"
temperature = response_json['main']['temp'] # 298.15
humidity = response_json['main']['humidity'] # 60
description = response_json['weather'][0]['description'] # ✅ "clear sky"
 # [0] needed - weather is a LIST
icon = response_json['weather'][0]['icon'] # "01d"
wind_speed = response_json['wind']['speed'] # 3.5

# ❌ Wrong access patterns:
response_json['weather']['description'] # ❌ weather is a list, not dict
response_json['main']['weather'] # ❌ wrong key
response_json['description']['weather'] # ❌ completely wrong order

# Safe access with .get() - avoids KeyError
temp = response_json.get('main', {}).get('temp', 0)

# Convert Kelvin to Celsius
temp_celsius = response_json['main']['temp'] - 273.15

6. `requests` Library - Complete Reference#

Installation & Import:#

pip install requests
import requests

All HTTP Methods:#

# GET
r = requests.get(url, params={'key': 'value'})

# POST
r = requests.post(url, json={'key': 'value'})
r = requests.post(url, data={'key': 'value'}) # form data

# PUT
r = requests.put(url, json={'key': 'value'})

# PATCH
r = requests.patch(url, json={'key': 'value'})

# DELETE
r = requests.delete(url)

# HEAD
r = requests.head(url)

Request Options:#

requests.get(
 url,
 params={'q': 'Delhi', 'appid': key}, # query parameters
 headers={
 'Authorization': f'Bearer {token}',
 'Content-Type': 'application/json',
 'User-Agent': 'MyApp/1.0'
 },
 json={'key': 'value'}, # request body (auto-sets Content-Type)
 data={'key': 'value'}, # form-encoded body
 timeout=30, # seconds before timeout
 verify=True, # SSL certificate verification
 auth=('user', 'pass'), # basic authentication
 cookies={'session': 'abc'} # send cookies
)

Response Object:#

r = requests.get(url)

# Status
r.status_code # 200, 404, etc.
r.ok # True if 200-299
r.raise_for_status() # raises exception if 4xx/5xx ✅

# Content
r.text # response as string
r.json() # parse JSON response ✅
r.content # response as bytes
r.headers # response headers dict
r.url # final URL (after redirects)
r.history # redirect history
r.elapsed # time taken for request
r.cookies # response cookies

Error Handling:#

import requests
from requests.exceptions import (
 RequestException,
 ConnectionError,
 Timeout,
 HTTPError
)

def safe_api_call(url, params):
 try:
 response = requests.get(url, params=params, timeout=30)
 response.raise_for_status() # raise for 4xx/5xx ✅
 return response.json()

 except requests.exceptions.Timeout:
 print("Request timed out")
 return None

 except requests.exceptions.ConnectionError:
 print("Connection failed")
 return None

 except requests.exceptions.HTTPError as e:
 print(f"HTTP error: {e.response.status_code}")
 if e.response.status_code == 401:
 print("Check your API key")
 elif e.response.status_code == 429:
 print("Rate limit exceeded, slow down")
 return None

 except requests.exceptions.RequestException as e:
 print(f"Request failed: {e}")
 return None

Sessions - Reuse Connection:#

# ✅ Use session for multiple requests to same API
# More efficient - reuses TCP connection

session = requests.Session()

# Set default headers for all requests
session.headers.update({
 'Authorization': f'Bearer {token}',
 'Content-Type': 'application/json'
})

# All requests use session settings
r1 = session.get(url1)
r2 = session.get(url2)
r3 = session.post(url3, json=data)

session.close() # or use context manager:

with requests.Session() as session:
 session.headers.update({'Authorization': f'Bearer {token}'})
 r = session.get(url)

7. API Pagination#

What is Pagination?#

API returns data in pages (not all at once)
Need to make multiple requests to get all data

Common Pagination Patterns:#

1. Page Number Pagination:#

def get_all_results(base_url, params):
 """Fetch all pages of results"""
 all_results = []
 page = 1

 while True:
 params['page'] = page
 params['per_page'] = 100

 response = requests.get(base_url, params=params)
 data = response.json()

 results = data.get('results', [])
 if not results:
 break # no more pages

 all_results.extend(results)
 page += 1

 time.sleep(0.5) # rate limiting

 return all_results

2. Cursor Pagination:#

def get_all_with_cursor(url, params):
 all_results = []
 cursor = None

 while True:
 if cursor:
 params['cursor'] = cursor

 response = requests.get(url, params=params)
 data = response.json()

 all_results.extend(data['items'])

 cursor = data.get('next_cursor')
 if not cursor:
 break # no more pages

 return all_results

3. Link Header Pagination (GitHub style):#

def get_all_github_results(url, headers):
 all_results = []

 while url:
 response = requests.get(url, headers=headers)
 all_results.extend(response.json())

 # GitHub puts next page URL in Link header
 link_header = response.headers.get('Link', '')
 next_url = None
 for part in link_header.split(','):
 if 'rel="next"' in part:
 next_url = part.split(';')[0].strip().strip('<>')
 url = next_url

 return all_results

8. Chrome DevTools - Network Tab for APIs#

What Network Tab Shows:#

Network Tab Features:
├── All HTTP requests made by page
├── Request details:
│ ├── Headers → request/response headers ✅
│ │ ├── Request URL
│ │ ├── Request Method (GET, POST)
│ │ ├── Status Code (200, 401, 429)
│ │ ├── Authorization header (API key location)
│ │ └── Content-Type
│ ├── Payload → request body (POST data)
│ ├── Response → actual response data ✅
│ │ └── JSON response body
│ └── Timing → request timing breakdown
│ ├── DNS lookup
│ ├── Connection time
│ ├── Time to First Byte (TTFB)
│ └── Content download
│
├── Filter by type:
│ └── XHR/Fetch → API calls specifically
│
└── Throttling → simulate slow network

Debugging API Issues:#

Problem: API call returns 401
→ Network tab → click request → Headers
→ Check: is Authorization header present?
→ Check: is API key format correct?
 Bearer token: "Authorization: Bearer sk-..."
 API key: "x-api-key: your-key"

Problem: Rate limiting (429)
→ Network tab → filter XHR
→ Look for pattern of 429 responses
→ Use Timing tab to see request frequency
→ Add sleep() between requests

Problem: Wrong data in response
→ Network tab → click request → Response tab
→ See exact JSON returned
→ Compare with expected structure

Complete API Workflow - Putting It Together:#

import requests
import time
import os
import json
from dotenv import load_dotenv

# Load environment variables
load_dotenv()

class WeatherAPIClient:
 def __init__(self):
 self.api_key = os.getenv('WEATHER_API_KEY') # ✅ env variable
 self.base_url = 'https://api.openweathermap.org/data/2.5'
 self.session = requests.Session()
 self.session.params = {'appid': self.api_key}

 def get_weather(self, city, max_retries=3):
 """Fetch weather with error handling and retry"""
 url = f'{self.base_url}/weather'

 for attempt in range(max_retries):
 try:
 response = self.session.get(
 url,
 params={'q': city},
 timeout=30
 )
 response.raise_for_status() # raise on 4xx/5xx

 data = response.json()

 # ✅ Parse nested JSON correctly
 return {
 'city': data['name'],
 'temp_c': data['main']['temp'] - 273.15,
 'humidity': data['main']['humidity'],
 'description': data['weather'][0]['description'] # ✅ [0]
 }

 except requests.exceptions.HTTPError as e:
 if e.response.status_code == 401:
 raise Exception("Invalid API key") # don't retry

 elif e.response.status_code == 429:
 wait = 2 ** attempt # exponential backoff
 print(f"Rate limited. Waiting {wait}s")
 time.sleep(wait)

 else:
 raise

 except requests.exceptions.Timeout:
 time.sleep(2 ** attempt)

 raise Exception(f"Failed after {max_retries} attempts")

 def get_multiple_cities(self, cities):
 """Fetch weather for multiple cities with rate limiting"""
 results = []

 for city in cities:
 try:
 data = self.get_weather(city)
 results.append(data)
 print(f"✅ Got weather for {city}")

 except Exception as e:
 print(f"❌ Failed for {city}: {e}")

 time.sleep(1) # ✅ rate limiting between requests

 return results

# Usage
client = WeatherAPIClient()
cities = ['Delhi', 'Mumbai', 'Chennai', 'Kolkata', 'Bangalore']
weather_data = client.get_multiple_cities(cities)

for city_data in weather_data:
 print(f"{city_data['city']}: {city_data['temp_c']:.1f}°C, {city_data['description']}")

HTTP & APIs - Quick Reference Card#

Status Codes:
 2xx → Success (200 OK, 201 Created, 204 No Content)
 3xx → Redirect (301 Permanent, 302 Temporary, 304 Cached)
 4xx → Client Error (400 Bad, 401 Auth ✅, 403 Forbidden,
 404 Not Found, 429 Rate Limit ✅)
 5xx → Server Error (500 Internal, 503 Unavailable)

Range Checks:
 200 <= status < 300 → success ✅
 300 <= status < 400 → redirect ✅

HTTP Methods:
 GET → retrieve (safe + idempotent) ✅
 POST → create/send (LLM inference) ✅
 PUT → replace (idempotent) ✅
 DELETE → remove (idempotent)
 PATCH → partial update

Rate Limiting:
 429 → rate limit exceeded
 Fix: time.sleep(1-2) between requests ✅
 Advanced: exponential backoff

API Security:
 ✅ os.getenv('API_KEY')
 ✅ .env file + python-dotenv
 ❌ Hard-code in script
 ❌ Commit to git

JSON Parsing:
 response.json() # parse response
 data['weather'][0]['description'] # nested + list ✅
 data.get('key', default) # safe access

6. Web Scraping

Sat, 06 Jan 2024 00:00:00 +0000

6. Web Scraping#

1. Web Scraping - Overview#

What is Web Scraping?#

Programmatically extracting data from websites
Automates what a human would do manually in a browser
Used for: price monitoring, research data collection, news aggregation, lead generation

General Workflow:#

1. Send HTTP request to URL
 ↓
2. Receive HTML response
 ↓
3. Parse HTML to find data
 ↓
4. Extract specific elements
 ↓
5. Clean and store data

Tools Available:#

Tool	Purpose	Best For
`requests`	Fetch HTML pages	Simple static pages
`BeautifulSoup`	Parse HTML/XML	Extracting data from HTML
`Scrapy`	Full scraping framework	Large-scale scraping
`Selenium`	Browser automation	Dynamic/JavaScript pages
`Playwright`	Modern browser automation	Modern JS-heavy pages
`httpx`	Async HTTP requests	Concurrent scraping

2. Ethical Scraping - `robots.txt`#

What is `robots.txt`?#

Text file at root of website: https://example.com/robots.txt
Tells web crawlers which paths are allowed or disallowed
Standard convention - not technically enforced but legally/ethically important

Reading `robots.txt`:#

# Example robots.txt
User-agent: * # applies to all bots
Disallow: /admin/ # don't scrape /admin/
Disallow: /private/ # don't scrape /private/
Allow: /public/ # explicitly allow /public/
Crawl-delay: 2 # wait 2 seconds between requests ✅

User-agent: Googlebot # specific rule for Google
Allow: / # Googlebot can access everything

User-agent: BadBot # block specific bot
Disallow: / # disallow everything

Checking `robots.txt` in Python:#

from urllib.robotparser import RobotFileParser

def can_scrape(url, user_agent='*'):
 """Check if URL can be scraped according to robots.txt"""
 # Parse robots.txt
 rp = RobotFileParser()
 
 # Get base URL
 from urllib.parse import urlparse
 parsed = urlparse(url)
 robots_url = f"{parsed.scheme}://{parsed.netloc}/robots.txt"
 
 rp.set_url(robots_url)
 rp.read()
 
 # Check if allowed
 allowed = rp.can_fetch(user_agent, url)
 crawl_delay = rp.crawl_delay(user_agent)
 
 return allowed, crawl_delay

# Usage
url = 'https://example.com/products'
allowed, delay = can_scrape(url)

if allowed:
 response = requests.get(url)
 # process...
 if delay:
 time.sleep(delay) # ✅ respect crawl-delay
else:
 print(f"Scraping {url} is not allowed by robots.txt")

Ethical Scraping Checklist:#

✅ Parse robots.txt before scraping
✅ Follow disallowed paths
✅ Implement crawl-delay between requests
✅ Identify your bot in User-Agent header
✅ Don't overload servers
✅ Check website's Terms of Service
✅ Cache responses to avoid redundant requests
✅ Scrape during off-peak hours if large volume

❌ Ignore robots.txt even for academic purposes
❌ Treat robots.txt as optional guidance
❌ Make rapid-fire requests without delays
❌ Scrape personal/private data without consent
❌ Bypass authentication/paywalls

✅ Parse robots.txt, follow disallowed paths, implement crawl-delay - exam answer
❌ Ignore robots.txt if data is for academic purposes - wrong
❌ Check robots.txt only once per domain - insufficient
❌ Treat robots.txt as optional - unethical

3. `crawl-delay` Directive#

import time
import requests
from urllib.robotparser import RobotFileParser

rp = RobotFileParser()
rp.set_url('https://example.com/robots.txt')
rp.read()

# Get crawl delay
delay = rp.crawl_delay('*') # for all bots
delay = delay if delay else 1 # default 1 second

urls_to_scrape = ['https://example.com/page1',
 'https://example.com/page2',
 'https://example.com/page3']

for url in urls_to_scrape:
 if rp.can_fetch('*', url):
 response = requests.get(url)
 process(response)
 time.sleep(delay) # ✅ respect crawl-delay
 else:
 print(f"Skipping disallowed URL: {url}")

4. BeautifulSoup - HTML Parsing#

What is BeautifulSoup?#

Python library for parsing HTML and XML
Makes it easy to navigate, search, and extract data from HTML
Works with requests to fetch and parse web pages
❌ NOT for making HTTP requests (use requests for that)
❌ NOT for dynamic/JavaScript pages (use Selenium/Playwright)

Installation:#

pip install beautifulsoup4
pip install lxml # faster parser (optional)

Import:#

from bs4 import BeautifulSoup
import requests

5. `BeautifulSoup()` - Creating a Soup Object#

Basic HTML Parsing:#

import requests
from bs4 import BeautifulSoup

# Step 1: Fetch HTML
url = 'https://example.com'
response = requests.get(url)
html_content = response.text

# Step 2: Create soup object
soup = BeautifulSoup(html_content, 'html.parser') # ✅ built-in parser
soup = BeautifulSoup(html_content, 'lxml') # faster, needs lxml
soup = BeautifulSoup(html_content, 'html5lib') # most lenient

# Step 3: Extract data
title = soup.title.text
print(title)

Parsers Comparison:#

Parser	Speed	Leniency	Requires Install
`html.parser`	Medium	Medium	No (built-in)
`lxml`	Fast	Strict	Yes
`html5lib`	Slow	Most lenient	Yes

Parsing from String (for testing):#

html = """
<html>
<body>
 <h1 class="title">Data Science Tools</h1>
 <div id="content">
 <p class="intro">Welcome to TDS</p>
 <ul>
 <li>Python</li>
 <li>Pandas</li>
 <li>Git</li>
 </ul>
 </div>
 <table>
 <tr><th>Tool</th><th>Purpose</th></tr>
 <tr><td>Git</td><td>Version Control</td></tr>
 <tr><td>Docker</td><td>Containerization</td></tr>
 </table>
 <a href="https://example.com">Click here</a>
</body>
</html>
"""

soup = BeautifulSoup(html, 'html.parser')

6. `.find()` and `.find_all()` - Locating Elements#

`.find()` - Returns FIRST matching element:#

# Find by tag name
h1 = soup.find('h1') # first <h1>
div = soup.find('div') # first <div>
p = soup.find('p') # first <p>

# Find by CSS class
intro = soup.find('p', class_='intro') # <p class="intro">
title = soup.find('h1', class_='title') # <h1 class="title">

# Find by id
content = soup.find('div', id='content') # <div id="content">
content = soup.find(id='content') # same result

# Find by attribute
link = soup.find('a', href=True) # any <a> with href
link = soup.find('a', href='https://example.com')

# Find by multiple attributes
elem = soup.find('div', {'class': 'intro', 'id': 'main'})

# Shorthand (tag as attribute)
title = soup.h1 # first <h1>
first_p = soup.p # first <p>

`.find_all()` - Returns ALL matching elements:#

# Find all by tag
all_links = soup.find_all('a') # all <a> tags
all_paragraphs = soup.find_all('p') # all <p> tags
all_rows = soup.find_all('tr') # all table rows

# Find all by class
all_items = soup.find_all('li', class_='item')

# Find all by attribute
all_hrefs = soup.find_all('a', href=True)

# Limit results
first_3 = soup.find_all('p', limit=3)

# Find multiple tag types
all_headings = soup.find_all(['h1', 'h2', 'h3'])

# Find by regex
import re
all_h = soup.find_all(re.compile('^h[1-6]$')) # all heading tags

Navigating the Tree:#

# Parent/child navigation
div = soup.find('div', id='content')

# Children
children = list(div.children) # direct children
descendants = list(div.descendants) # all descendants

# First/last child
first = div.find() # first child element

# Parent
parent = div.parent # parent element

# Siblings
next_sib = div.next_sibling
prev_sib = div.previous_sibling
all_siblings = div.next_siblings

# Find within element (scoped search)
ul = soup.find('ul')
items = ul.find_all('li') # only li inside this ul

7. `.get_text()` - Extracting Text#

# Extract text from element
h1 = soup.find('h1')
text = h1.get_text() # "Data Science Tools"
text = h1.text # same result

# Get text with separator
div = soup.find('div', id='content')
text = div.get_text(separator='\n') # text with newlines
text = div.get_text(separator=' ') # text with spaces
text = div.get_text(strip=True) # strip whitespace ✅

# Get text from all matching elements
all_li = soup.find_all('li')
items = [li.get_text(strip=True) for li in all_li]
# ['Python', 'Pandas', 'Git']

# Get all text on page
all_text = soup.get_text()
all_text = soup.get_text(separator='\n', strip=True)

# Clean text
import re
clean = re.sub(r'\s+', ' ', all_text).strip()

8. `.get('href')` - Extracting Attributes#

# Get attribute value
link = soup.find('a')
href = link.get('href') # ✅ safe (returns None if missing)
href = link['href'] # raises KeyError if missing

# Get any attribute
img = soup.find('img')
src = img.get('src')
alt = img.get('alt', '') # with default value
width = img.get('width')

# Check if attribute exists
if link.has_attr('href'):
 print(link['href'])

# Get all attributes as dict
attrs = link.attrs # {'href': 'url', 'class': ['link']}

# Get all links on page
all_links = soup.find_all('a', href=True)
urls = [link.get('href') for link in all_links]

# Resolve relative URLs
from urllib.parse import urljoin
base_url = 'https://example.com'
absolute_urls = [urljoin(base_url, url) for url in urls]

Complete BeautifulSoup Example - Scraping a Table:#

import requests
from bs4 import BeautifulSoup
import pandas as pd
import time

def scrape_table(url):
 """
 Scrape HTML table from webpage
 ✅ Follows ethical scraping practices
 """
 # Set User-Agent to identify bot
 headers = {
 'User-Agent': 'DataScienceBot/1.0 (research purposes)'
 }

 # Fetch page
 response = requests.get(url, headers=headers, timeout=30)
 response.raise_for_status()

 # Parse HTML
 soup = BeautifulSoup(response.text, 'html.parser')

 # Find table
 table = soup.find('table')
 if not table:
 return None

 # Extract headers
 headers_row = table.find('tr')
 headers = [th.get_text(strip=True)
 for th in headers_row.find_all('th')]

 # Extract rows
 rows = []
 for tr in table.find_all('tr')[1:]: # skip header row
 cells = [td.get_text(strip=True)
 for td in tr.find_all('td')]
 if cells:
 rows.append(cells)

 # Convert to DataFrame
 df = pd.DataFrame(rows, columns=headers)
 return df

def scrape_multiple_pages(base_url, pages):
 """Scrape multiple pages with rate limiting"""
 all_data = []

 for page in range(1, pages + 1):
 url = f'{base_url}?page={page}'

 try:
 df = scrape_table(url)
 if df is not None:
 all_data.append(df)
 print(f"✅ Scraped page {page}")
 except Exception as e:
 print(f"❌ Failed page {page}: {e}")

 time.sleep(2) # ✅ rate limiting

 return pd.concat(all_data, ignore_index=True)

Complete BeautifulSoup Reference:#

# CSS Selectors (alternative to find/find_all)
soup.select('div.content') # <div class="content">
soup.select('#main') # id="main"
soup.select('table tr td') # nested elements
soup.select('a[href]') # <a> with href attribute
soup.select('p.intro, p.outro') # multiple selectors

# First match with CSS selector
soup.select_one('h1.title')

# Practical extraction patterns:

# 1. All links
links = [(a.get_text(strip=True), a.get('href'))
 for a in soup.find_all('a', href=True)]

# 2. All images
images = [(img.get('alt', ''), img.get('src'))
 for img in soup.find_all('img', src=True)]

# 3. Table to DataFrame
import pandas as pd
tables = pd.read_html(str(soup)) # ✅ pandas can parse HTML tables
df = tables[0] # first table

# 4. Specific div content
content = soup.find('div', class_='article-body')
paragraphs = [p.get_text(strip=True)
 for p in content.find_all('p')]

# 5. Meta tags
meta_desc = soup.find('meta', attrs={'name': 'description'})
description = meta_desc.get('content') if meta_desc else ''

# 6. JSON-LD structured data
import json
script = soup.find('script', type='application/ld+json')
if script:
 data = json.loads(script.string)

# 7. Next page link (pagination)
next_page = soup.find('a', {'rel': 'next'})
next_url = next_page.get('href') if next_page else None

9. Scrapy - Full Web Scraping Framework#

What is Scrapy?#

Full-featured, asynchronous web scraping framework
Built for large-scale scraping
Handles: requests, parsing, pipelines, rate limiting automatically
More complex than BeautifulSoup but much more powerful

When to Use Scrapy vs BeautifulSoup:#

Scenario	Use
Simple, one-off scraping	BeautifulSoup + requests
Large-scale, many pages	Scrapy
Need JavaScript	Selenium/Playwright
Need data pipelines	Scrapy
Quick data extraction	BeautifulSoup

Basic Scrapy Spider:#

import scrapy

class ProductSpider(scrapy.Spider):
 name = 'products'
 start_urls = ['https://example.com/products']

 # Respect robots.txt
 custom_settings = {
 'ROBOTSTXT_OBEY': True, # ✅ obey robots.txt
 'DOWNLOAD_DELAY': 2, # ✅ 2 second delay
 'CONCURRENT_REQUESTS': 4, # max concurrent requests
 'USER_AGENT': 'DataBot/1.0'
 }

 def parse(self, response):
 # Extract data from page
 for product in response.css('div.product'):
 yield {
 'name': product.css('h2::text').get(),
 'price': product.css('.price::text').get(),
 'url': product.css('a::attr(href)').get()
 }

 # Follow next page link
 next_page = response.css('a.next-page::attr(href)').get()
 if next_page:
 yield response.follow(next_page, self.parse)

10. `requests` + BeautifulSoup - Combined Pattern#

Standard Workflow:#

import requests
from bs4 import BeautifulSoup
import time
import os

def scrape_page(url, session=None):
 """
 Standard requests + BeautifulSoup workflow
 """
 requester = session or requests

 response = requester.get(
 url,
 headers={'User-Agent': 'Research Bot 1.0'},
 timeout=30
 )
 response.raise_for_status()

 soup = BeautifulSoup(response.text, 'html.parser')
 return soup

def extract_articles(soup):
 """Extract article data from parsed HTML"""
 articles = []

 for article in soup.find_all('article'):
 title = article.find('h2')
 link = article.find('a')
 date = article.find('time')
 summary = article.find('p', class_='summary')

 articles.append({
 'title': title.get_text(strip=True) if title else '',
 'url': link.get('href') if link else '',
 'date': date.get('datetime') if date else '',
 'summary': summary.get_text(strip=True) if summary else ''
 })

 return articles

# Full pipeline
def scrape_news_site(base_url, num_pages=5):
 all_articles = []

 with requests.Session() as session:
 session.headers.update({
 'User-Agent': 'DataScience Research Bot 1.0'
 })

 for page in range(1, num_pages + 1):
 url = f'{base_url}/news?page={page}'

 try:
 soup = scrape_page(url, session)
 articles = extract_articles(soup)
 all_articles.extend(articles)
 print(f"Page {page}: {len(articles)} articles")
 except Exception as e:
 print(f"Failed page {page}: {e}")

 time.sleep(2) # ✅ rate limiting

 return all_articles

11. Handling Dynamic Pages - Selenium & Playwright#

When is JavaScript Needed?#

Static page (BeautifulSoup works):
→ HTML is fully rendered in response
→ Data visible in page source (Ctrl+U)

Dynamic page (need Selenium/Playwright):
→ Data loaded by JavaScript after page loads
→ Data NOT in page source
→ Requires button clicks, scrolling, waiting
→ Examples: SPAs, infinite scroll, login-required pages

Selenium Basic Usage:#

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup
import time

# Setup driver (Chrome)
options = webdriver.ChromeOptions()
options.add_argument('--headless') # run without browser window
options.add_argument('--no-sandbox')

driver = webdriver.Chrome(options=options)

try:
 # Navigate to page
 driver.get('https://example.com')

 # Wait for element to load
 wait = WebDriverWait(driver, 10)
 element = wait.until(
 EC.presence_of_element_located((By.CLASS_NAME, 'product'))
 )

 # Scroll to load more content
 driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")
 time.sleep(2)

 # Click button
 button = driver.find_element(By.ID, 'load-more')
 button.click()
 time.sleep(2)

 # Get rendered HTML and parse with BeautifulSoup
 html = driver.page_source
 soup = BeautifulSoup(html, 'html.parser')

 # Extract data normally
 products = soup.find_all('div', class_='product')
 for product in products:
 print(product.get_text(strip=True))

finally:
 driver.quit() # always close browser

Playwright (Modern Alternative):#

from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup

with sync_playwright() as p:
 browser = p.chromium.launch(headless=True)
 page = browser.new_page()

 page.goto('https://example.com')
 page.wait_for_selector('.product') # wait for element

 # Get content
 html = page.content()
 soup = BeautifulSoup(html, 'html.parser')

 browser.close()

12. Respecting Rate Limits in Scraping#

import time
import random
import requests
from urllib.robotparser import RobotFileParser

class EthicalScraper:
 """
 Scraper that respects robots.txt and rate limits
 """
 def __init__(self, base_url, delay=2):
 self.base_url = base_url
 self.delay = delay
 self.session = requests.Session()
 self.session.headers.update({
 'User-Agent': 'ResearchBot/1.0 (academic research)'
 })

 # Load robots.txt
 self.robot_parser = RobotFileParser()
 self.robot_parser.set_url(f'{base_url}/robots.txt')
 self.robot_parser.read()

 # Get crawl delay from robots.txt
 robot_delay = self.robot_parser.crawl_delay('*')
 if robot_delay:
 self.delay = max(self.delay, robot_delay) # use larger delay

 def fetch(self, url, retries=3):
 """Fetch URL with rate limiting and error handling"""
 # Check robots.txt
 if not self.robot_parser.can_fetch('*', url):
 print(f"Disallowed by robots.txt: {url}")
 return None

 for attempt in range(retries):
 try:
 response = self.session.get(url, timeout=30)
 response.raise_for_status()

 # ✅ Rate limiting - respect crawl delay
 # Add random jitter to avoid being detected as bot
 sleep_time = self.delay + random.uniform(0, 1)
 time.sleep(sleep_time)

 return response.text

 except requests.exceptions.HTTPError as e:
 if e.response.status_code == 429: # rate limited
 time.sleep(60) # wait 1 minute
 elif e.response.status_code == 404:
 return None # page not found
 else:
 time.sleep(2 ** attempt) # backoff

 except requests.exceptions.RequestException:
 time.sleep(2 ** attempt)

 return None

 def scrape(self, urls):
 """Scrape multiple URLs ethically"""
 results = []

 for url in urls:
 html = self.fetch(url)
 if html:
 from bs4 import BeautifulSoup
 soup = BeautifulSoup(html, 'html.parser')
 results.append({'url': url, 'soup': soup})

 return results

Web Scraping - Quick Reference Card#

Tool Selection:
 Static HTML page → requests + BeautifulSoup ✅
 JavaScript page → Selenium or Playwright
 Large-scale scraping → Scrapy
 Simple HTTP → requests alone

BeautifulSoup:
 soup = BeautifulSoup(html, 'html.parser')
 soup.find('tag') → first element
 soup.find('tag', class_='name') → by class
 soup.find(id='name') → by id
 soup.find_all('tag') → all elements
 element.get_text(strip=True) → extract text ✅
 element.get('href') → extract attribute ✅
 soup.select('div.class > p') → CSS selector

Ethical Scraping:
 ✅ Check robots.txt first
 ✅ Follow crawl-delay
 ✅ Add time.sleep() between requests
 ✅ Set User-Agent header
 ❌ Ignore robots.txt
 ❌ Rapid-fire requests

Common Patterns:
 # Fetch + parse
 response = requests.get(url)
 soup = BeautifulSoup(response.text, 'html.parser')

 # Extract all links
 links = [a.get('href') for a in soup.find_all('a', href=True)]

 # Extract table
 df = pd.read_html(response.text)[0]

 # Extract all text
 text = soup.get_text(separator='\n', strip=True)

✅ Section 6 done! Proceed to Section 7: Python Error Handling & Logging?

7. Python Error Handling & Logging

Sun, 07 Jan 2024 00:00:00 +0000

7. Python Error Handling & Logging#

1. `try-except-finally` - Overview#

Used to handle runtime errors gracefully and ensure cleanup code always runs
Most robust and maintainable solution for unexpected errors in data pipelines

2. Execution Order - No Exception:#

try → finally
# except block is SKIPPED if no exception occurs

3. Execution Order - Exception Occurs:#

try → except → finally

4. `finally` Block:#

ALWAYS runs regardless of whether exception occurred or not
Used for: closing files, DB connections, releasing resources

try:
 result = risky_operation()
except ValueError as e:
 print(f"Error: {e}")
finally:
 cleanup() # ALWAYS runs ✅

5. `else` Block:#

Runs only when NO exception occurs

try:
 result = risky_operation()
except ValueError:
 print("Error")
else:
 print("Success") # only if no exception
finally:
 cleanup() # always

6. Specific Exception Types:#

Always use specific exception types over generic ones

except ValueError # wrong value type
except FileNotFoundError # file doesn't exist
except KeyError # dictionary key missing
except TypeError # wrong data type
except ZeroDivisionError # division by zero
except Exception # catch-all (use sparingly)

7. Generic `Exception`:#

Catches everything - use only as last resort
Hides bugs by catching unexpected errors silently
✅ Always prefer specific exceptions

8. Common Wrong Answers:#

❌ try-catch-finally → Java/JavaScript syntax, NOT Python
❌ if-else-finally → not valid Python
❌ try only → no cleanup, no error handling
❌ Using only generic try-except → too broad, misses specific handling

9. `logging` Module:#

Better than print() for production code
Records errors with timestamps, severity levels

import logging

logging.debug("Debug info") # development details
logging.info("Pipeline started") # general info
logging.warning("Missing data") # something unexpected
logging.error("API call failed") # serious problem
logging.critical("DB crashed") # system failure

Log Levels (low → high):#

DEBUG < INFO < WARNING < ERROR < CRITICAL

10. Error Handling in Data Pipelines:#

# ✅ Exam-relevant pattern
try:
 df = pd.read_csv('data.csv')
except FileNotFoundError:
 logging.error("Data file missing")
except pd.errors.EmptyDataError:
 logging.error("File is empty")
finally:
 logging.info("Pipeline step complete")

8. Python & pandas - Core Operations

Mon, 08 Jan 2024 00:00:00 +0000

8. Python & pandas - Core Operations#

1. Detecting Missing Values:#

df.isnull().sum() # count NaN per column
df.isnull().sum() / len(df) * 100 # percentage missing

2. `df.count()` - Count Non-Null Values:#

df.count() # non-null count per column ✅
df['col'].count() # for single column

❌ df.notnull() → returns boolean mask, not count
❌ df.describe() → summary stats, not just count

3. `df.dropna()` - Remove Missing Values:#

df.dropna() # drop rows with ANY NaN
df.dropna(subset=['review_text']) # drop rows where specific column is NaN ✅
df.dropna(how='all') # drop only if ALL values are NaN

✅ dropna(subset=['col']) - exam answer for targeted column NaN removal
❌ Replace all NaN with “missing” → pollutes text analysis
❌ Delete entire column → loses all data

4. `df.fillna()` - Fill Missing Values:#

df.fillna(0) # fill with constant
df['col'].fillna(df['col'].mean()) # fill with mean
df['col'].fillna(df['col'].median()) # fill with median
df.fillna(method='ffill') # forward fill (time series)
df.fillna(method='bfill') # backward fill

5. When to Use Which Fill Strategy:#

Strategy	When
Drop rows	Text analysis, small % missing
Fill mean	Normally distributed numeric
Fill median	Skewed data, outliers present
Fill mode	Categorical data
Forward fill	Time series, ordered data

6. `column.nunique()` - Count Distinct Values:#

df['col'].nunique() # number of unique values ✅
df['col'].unique() # array of unique values
df['col'].value_counts() # count of each unique value

✅ nunique() - most efficient for just counting distinct values
❌ len(set(column)) → works but memory inefficient for large data
❌ column.count() → counts non-null, NOT unique

7. Value Inspection Methods:#

Method	Returns
`nunique()`	Integer count of unique values ✅
`unique()`	Array of all unique values
`value_counts()`	Frequency of each unique value
`count()`	Count of non-null values

8. `groupby().mean()` - Aggregation:#

df.groupby('cuisine')['rating'].mean() # average rating per cuisine ✅

❌ df.sort_values('cuisine') → sorts, doesn’t aggregate
❌ df.filter('cuisine') → not valid syntax
❌ df.merge('cuisine', 'rating') → not valid syntax

9. All Aggregation Functions:#

df.groupby('col')['val'].mean() # average
df.groupby('col')['val'].sum() # total
df.groupby('col')['val'].count() # count
df.groupby('col')['val'].max() # maximum
df.groupby('col')['val'].min() # minimum
df.groupby('col')['val'].median() # median
df.groupby('col')['val'].std() # standard deviation

10. `drop_duplicates()` - Remove Duplicates:#

df.drop_duplicates() # all columns
df.drop_duplicates(subset=['col']) # specific column ✅
df['col'].drop_duplicates().values # memory-efficient unique values ✅

11. Correlation Analysis:#

df.corr() # correlation matrix for all columns
df['col1'].corr(df['col2']) # correlation between two columns

# pivot_table + corr for customer similarity
df.pivot_table(
 index='customer_id',
 columns='product_category',
 values='purchase_amount'
).fillna(0).corr() # ✅ exam answer

12. Sorting:#

df.sort_values('col') # ascending
df.sort_values('col', ascending=False) # descending
df.sort_values(['col1', 'col2']) # multiple columns

13. Merging DataFrames:#

pd.merge(df1, df2, on='key') # inner join (default)
pd.merge(df1, df2, on='key', how='left') # left join
pd.merge(df1, df2, on='key', how='outer') # outer join
pd.concat([df1, df2]) # stack vertically

14. Filtering Rows:#

df[df['rating'] < 3.0] # single condition ✅
df[(df['rating'] < 3) & (df['city'] == 'NYC')] # multiple conditions
df[df['col'].isin(['val1', 'val2'])] # filter by list
df[df['col'].str.contains('pattern')] # string filter

15. String Operations on Columns:#

df['col'].str.lower() # lowercase
df['col'].str.upper() # uppercase
df['col'].str.strip() # remove whitespace
df['col'].str.replace('a', 'b') # replace
df['col'].str.split(',') # split
df['col'].str.contains('pat') # boolean mask

16. DataFrame Exploration:#

df.head() # first 5 rows
df.tail() # last 5 rows
df.info() # column types + non-null counts
df.describe() # summary statistics
df.shape # (rows, columns)
df.dtypes # data type of each column
df.columns # column names

17. Data Types:#

df.dtypes # check all types
df['col'].astype('int') # convert to integer
df['col'].astype('float') # convert to float
df['col'].astype('str') # convert to string
df['col'].astype('category') # convert to category (memory efficient)
pd.to_datetime(df['date_col']) # convert to datetime

18. Reading & Writing Data:#

pd.read_csv('file.csv')
pd.read_json('file.json')
pd.read_parquet('file.parquet')
pd.read_excel('file.xlsx')
pd.read_sql(query, connection)

df.to_csv('output.csv', index=False)
df.to_json('output.json')
df.to_parquet('output.parquet')
df.to_excel('output.xlsx', index=False)

19. `collections.Counter`:#

from collections import Counter

# Count frequencies
Counter(['a', 'b', 'a', 'c']) # {'a': 2, 'b': 1, 'c': 1}
Counter.most_common(5) # top 5 most frequent ✅

# Count URL segments in log analysis
Counter(
 url.split('/')[2]
 for url in urls
 if len(url.split('/')) > 2
)

Quick Reference#

Missing values:
 df.isnull().sum() → count NaN
 df.dropna(subset=['col']) → remove targeted NaN ✅
 df.fillna(value) → fill NaN

Unique values:
 df['col'].nunique() → count distinct ✅
 df['col'].unique() → array of unique
 df['col'].value_counts() → frequency table

Aggregation:
 df.groupby('col')['val'].mean() ✅

Duplicates:
 df.drop_duplicates(subset=['col']) ✅

Exception handling:
 No exception → try → finally ✅
 Exception → try → except → finally ✅
 finally → ALWAYS runs ✅
 try-catch → Java, NOT Python ❌

9. Python datetime & String Operations

Tue, 09 Jan 2024 00:00:00 +0000

9. Python datetime & String Operations#

1. `datetime.strptime()` - Parse String to Datetime#

Converts a string → datetime object
Need to specify exact format matching the string

from datetime import datetime
timestamp = datetime.strptime("14/Dec/2024:16:45:11 -0500", "%d/%b/%Y:%H:%M:%S %z")

2. `%d/%b/%Y:%H:%M:%S %z` - Apache Log Format#

Most exam-relevant format string
Used to parse Apache/Nginx log timestamps

[14/Dec/2024:16:45:11 -0500]
 %d %b %Y %H %M %S %z

3. `%z` - Timezone Offset (Critical!)#

Parses timezone offset like -0500, +0530
Without %z → parsing fails or ignores timezone
✅ Always include %z for Apache log timestamps
❌ %Z → timezone NAME (EST, UTC) - different from offset

4. `timestamp.hour` - Extract Hour:#

timestamp.hour # 0-23 integer
timestamp.minute # 0-59
timestamp.second # 0-59
timestamp.date() # date part only
timestamp.time() # time part only

5. `timestamp.weekday()` - Day of Week:#

timestamp.weekday() # returns integer

# Mapping:
0 → Monday
1 → Tuesday
2 → Wednesday
3 → Thursday
4 → Friday
5 → Saturday
6 → Sunday

✅ weekday() == 6 → Sunday (exam answer JAN_FN)
✅ weekday() == 5 → Saturday (exam answer JAN_AN)
✅ weekday() == 0 → Monday
❌ timestamp.dayname() → method doesn’t exist
❌ timestamp.strftime('%A') == 'Sunday' → works but less efficient

6. Time Range Filtering - Core Logic:#

Key Rule:#

For range X:00 to Y:59
→ X <= timestamp.hour <= Y

For range X:00 to Y:00 exclusive
→ X <= timestamp.hour < Y+1

Exam Scenarios:#

# 12:00 to 15:59 (includes 12,13,14,15)
12 <= timestamp.hour <= 15 # ✅

# 16:00 to 18:59 (includes 16,17,18)
16 <= timestamp.hour < 19 # ✅ exam answer JAN_FN

# 14:00 to 16:59 (includes 14,15,16)
14 <= timestamp.hour < 17 # ✅ exam answer JAN_AN

Common Wrong Answers:#

16 <= timestamp.hour <= 16 # ❌ only hour 16
16 < timestamp.hour < 19 # ❌ excludes 16:xx entries
timestamp.hour in range(16, 20) # ❌ includes 19:xx entries

7. Format Codes - Complete Reference:#

Code	Meaning	Example
`%d`	Day of month (zero-padded)	01–31
`%m`	Month as number	01–12
`%b`	Abbreviated month name	Jan, Dec
`%B`	Full month name	January
`%Y`	4-digit year	2024
`%y`	2-digit year	24
`%H`	Hour 24-hr	00–23
`%I`	Hour 12-hr	01–12
`%M`	Minutes	00–59
`%S`	Seconds	00–59
`%z`	UTC offset	-0500
`%Z`	Timezone name	EST, UTC
`%A`	Full weekday name	Monday
`%a`	Abbreviated weekday	Mon
`%p`	AM/PM	AM, PM

8. String Splitting - Request Field Parsing#

Apache Request Field:#

"GET /checkout/payment HTTP/1.1"
 [0] [1] [2]

request = "GET /checkout/payment HTTP/1.1"
parts = request.split(' ')

method = parts[0] # 'GET'
url = parts[1] # '/checkout/payment' ✅ exam answer
protocol = parts[2] # 'HTTP/1.1' ✅ exam answer

✅ request.split(' ')[1] → URL (exam answer JAN_FN Q306)
✅ request.split(' ')[2] → Protocol (exam answer JAN_FN Q305)
❌ request.split('/')[1] → splits by slash, gives wrong result
❌ request.split(' ')[0] → gives HTTP method, not URL

9. URL Path Matching:#

url = '/tamilmp3/song.mp3'

url.startswith('/tamilmp3/') # ✅ prefix match (exam answer)
url.endswith('.mp3') # suffix match

# ❌ Wrong approaches:
'/tamilmp3/' in url # matches anywhere, not just start
url == '/tamilmp3/' # exact match only, misses sub-paths
url.endswith('/tamilmp3/') # checks end, not start

10. String Methods - Complete Reference:#

Case Operations:#

s.lower() # 'HELLO' → 'hello'
s.upper() # 'hello' → 'HELLO'
s.title() # 'hello world' → 'Hello World'
s.capitalize() # 'hello world' → 'Hello world'
s.swapcase() # 'Hello' → 'hELLO'

Whitespace:#

s.strip() # remove leading + trailing whitespace
s.lstrip() # remove leading only
s.rstrip() # remove trailing only

Search & Check:#

s.startswith('pre') # True/False
s.endswith('suf') # True/False
'sub' in s # True/False
s.find('sub') # index of first occurrence (-1 if not found)
s.index('sub') # index (raises ValueError if not found)
s.count('sub') # count occurrences

Replace & Split:#

s.replace('old', 'new') # replace all occurrences
s.split(',') # split by delimiter → list
s.split() # split by whitespace
','.join(['a','b','c']) # join list → 'a,b,c'

Type Checking:#

s.isdigit() # all digits?
s.isalpha() # all letters?
s.isalnum() # alphanumeric?
s.isnumeric() # numeric?
s.isspace() # all whitespace?
s.islower() # all lowercase?
s.isupper() # all uppercase?

Format:#

f"Hello {name}" # f-string ✅ most common
"Hello {}".format(name) # .format()
"Hello %s" % name # % operator (old style)
s.zfill(5) # '42' → '00042' (pad with zeros)
s.ljust(10), s.rjust(10) # left/right align
s.center(10) # center align

11. `re` Module - Regular Expressions#

What is Regex?#

Pattern matching for strings
Used heavily in log parsing

Core Functions:#

import re

re.search(pattern, string) # find first match anywhere
re.match(pattern, string) # match only at START
re.findall(pattern, string) # return ALL matches as list
re.sub(pattern, replacement, string) # replace matches
re.split(pattern, string) # split by pattern

Common Patterns:#

\d # any digit (0-9)
\d+ # one or more digits
\w # word character (letter, digit, underscore)
\s # whitespace
. # any character except newline
^ # start of string
$ # end of string
[A-Z] # any uppercase letter
[0-9] # any digit
(a|b) # a or b
* # zero or more
+ # one or more
? # zero or one
{3} # exactly 3
{2,5} # between 2 and 5

Log-Relevant Patterns:#

# Extract IP address
re.findall(r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}', log_line)

# Extract HTTP method
re.search(r'"(GET|POST|PUT|DELETE)', log_line)

# Extract status code
re.search(r'" (\d{3}) ', log_line)

# Extract timestamp
re.search(r'\[(\d{2}/\w+/\d{4}:\d{2}:\d{2}:\d{2})', log_line)

# Match time pattern for grep
re.compile(r'(0[89]|1[0-5]):[0-5][0-9]') # 08:xx to 15:xx

datetime - Additional Useful Operations:#

from datetime import datetime, timedelta

# Current time
now = datetime.now()
utc_now = datetime.utcnow()

# Date arithmetic
tomorrow = now + timedelta(days=1)
last_week = now - timedelta(weeks=1)
three_hours_later = now + timedelta(hours=3)

# Compare timestamps
ts1 > ts2 # is ts1 after ts2?
ts1 < ts2 # is ts1 before ts2?

# Convert to string
timestamp.strftime("%Y-%m-%d %H:%M:%S")

# pandas datetime
pd.to_datetime(df['col']) # parse column
pd.to_datetime(df['col'], format='%d/%b/%Y:%H:%M:%S %z')
df['col'].dt.hour # extract hour
df['col'].dt.weekday # day of week
df['col'].dt.tz_convert('UTC') # convert timezone

Quick Reference#

Apache log format string:
 "%d/%b/%Y:%H:%M:%S %z" ✅

Weekday numbers:
 0=Mon, 1=Tue, 2=Wed, 3=Thu, 4=Fri, 5=Sat, 6=Sun

Time range check:
 12:00-15:59 → 12 <= hour <= 15 ✅
 16:00-18:59 → 16 <= hour < 19 ✅
 14:00-16:59 → 14 <= hour < 17 ✅

Request field splitting:
 split(' ')[0] → method
 split(' ')[1] → URL ✅
 split(' ')[2] → protocol ✅

URL matching:
 url.startswith('/path/') ✅
 '/path/' in url ❌ (matches anywhere)

10. Apache Log Analysis

Wed, 10 Jan 2024 00:00:00 +0000

10. Apache Log Analysis#

1. Apache Log Format - Structure#

Standard Log Entry:#

IP RemoteUser AuthUser [Timestamp] "Method URL Protocol" Status Size "Referrer" "UserAgent" VHost ServerIP

Example Entry:#

203.0.113.7 - - [14/Dec/2024:16:45:11 -0500] "GET /index.html HTTP/1.1" 200 3500 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 15_0)" shop.com 192.168.1.100

Field Breakdown:#

Field	Example	Notes
IP	`203.0.113.7`	Visitor IP
RemoteUser	`-`	Typically `-`
AuthUser	`-`	Typically `-`
Timestamp	`[14/Dec/2024:16:45:11 -0500]`	Includes timezone
Method	`GET`	HTTP method
URL	`/index.html`	Path visited
Protocol	`HTTP/1.1`	HTTP version
Status	`200`	HTTP status code
Size	`3500`	Response bytes
Referrer	`"-"`	Where request came from
UserAgent	`"Mozilla/5.0..."`	Browser/device info
VHost	`shop.com`	Virtual host
ServerIP	`192.168.1.100`	Server IP

2. Valid Log Entry Rules#

Valid Entry Must Have:#

✅ Numeric HTTP status code (200, 404)
✅ Valid HTTP method (GET, POST, PUT, DELETE)
✅ Valid protocol string (HTTP/1.1, HTTP/2.0)

Invalid Entries:#

"POST /image.jpg INVALID" ❌ invalid protocol
"PUT /image.jpg HTTP/1.1" OK ❌ status must be numeric not "OK"

3. Log Filtering Fields#

To filter “POST requests for /images/ from 15:00–18:00 on Mondays” need:#

✅ Time → for hour AND day-of-week filtering
✅ Method → check POST
✅ URL → check /images/ path

NOT needed for this filter:#

❌ Status, Size, Referrer, Server → irrelevant to this filter

4. Mobile Detection from UserAgent#

# ✅ Correct - check multiple mobile indicators
mobile_indicators = ['mobile', 'iphone', 'android', 'ipad']
is_mobile = any(
 indicator in user_agent.lower()
 for indicator in mobile_indicators
)

# ❌ Wrong - too narrow
'Mobile' in user_agent # case sensitive
user_agent.startswith('Mozilla/5.0 (Mobile') # too specific
'webkit' in user_agent.lower() # webkit used by desktop too

5. Counting URL Segments - `collections.Counter`#

from collections import Counter

# Count checkout transactions by product category
# URLs like: /checkout/electronics/, /checkout/clothing/
Counter(
 url.split('/')[2]
 for url in filtered_checkout_urls
 if len(url.split('/')) > 2
)
# Result: {'electronics': 45, 'clothing': 32, 'food': 18}

6. Validate Filters Independently - Debugging#

Systematic Approach:#

# Validate each filter step by step
all_entries = parse_log('access.log')

# Step 1: Check time filter
time_filtered = [e for e in all_entries
 if 12 <= e['timestamp'].hour <= 15]
print(f"After time filter: {len(time_filtered)}")

# Step 2: Check method filter
method_filtered = [e for e in time_filtered
 if e['method'] == 'POST']
print(f"After method filter: {len(method_filtered)}")

# Step 3: Check URL filter
url_filtered = [e for e in method_filtered
 if e['url'].startswith('/checkout/')]
print(f"After URL filter: {len(url_filtered)}")

# Step 4: Check status filter
final = [e for e in url_filtered
 if 200 <= e['status'] < 300]
print(f"Final count: {len(final)}")

✅ Validate each filter independently - exam answer
❌ Assume data corruption → wrong first assumption
❌ Reduce dataset randomly → loses data integrity

7. “Any of These Could Be the Reason”#

If log count appears unexpectedly high → any filter could be wrong:
- Wrong weekday number
- Wrong hour range boundary
- Missing URL path filter
- Wrong HTTP method filter
✅ “Any of these could be the reason” - exam answer (JAN_FN Q310, JAN_AN Q429)

8. Double Filter for 404 Count#

# ✅ Must apply BOTH filters before counting
filtered = [
 e for e in entries
 if e['url'].startswith('/error/') # URL filter
 and e['status'] == 404 # status filter
]
count = len(filtered)

# ❌ Wrong - counts everything in filtered_entries
# without applying the second filter (status==404)
len(filtered_entries) # if filtered_entries only has URL filter applied

✅ “None of these” - exam answer (JAN_AN Q428) when options show incomplete filtering

9. Reading Compressed Logs#

import gzip

# ✅ Read gzipped log file
with gzip.open('access.log.gz', 'rt') as f:
 for line in f: # generator - memory efficient
 process(line)

10. Generator Pattern for Large Logs#

# ✅ Memory efficient - one line at a time
with open('access.log') as f:
 for line in f:
 process(line)

# ❌ Loads entire file into memory
lines = open('access.log').readlines()

Complete Log Parsing - Minimal Reference:#

import gzip
from datetime import datetime

def parse_log_line(line):
 """Parse single Apache log line"""
 import shlex
 try:
 parts = shlex.split(line) # handles quoted fields correctly
 return {
 'ip': parts[0],
 'timestamp': datetime.strptime(
 parts[3].strip('[]'),
 '%d/%b/%Y:%H:%M:%S %z' # ✅ exam format
 ),
 'method': parts[5], # split(' ')[0] of request
 'url': parts[6], # split(' ')[1] of request ✅
 'protocol': parts[7], # split(' ')[2] of request ✅
 'status': int(parts[8]),
 'size': parts[9],
 'user_agent': parts[11]
 }
 except:
 return None

def analyze_log(filepath):
 results = []
 
 # Handle both gzipped and plain files
 opener = gzip.open if filepath.endswith('.gz') else open
 
 with opener(filepath, 'rt') as f:
 for line in f: # ✅ generator pattern
 entry = parse_log_line(line.strip())
 if entry:
 results.append(entry)
 
 return results

Common Log Analysis Patterns:#

entries = analyze_log('access.log.gz')

# Filter by time range
peak_hour = [e for e in entries
 if 12 <= e['timestamp'].hour <= 15] # ✅

# Filter by weekday
mondays = [e for e in entries
 if e['timestamp'].weekday() == 0] # ✅ 0=Monday

# Filter POST to /checkout/
checkouts = [e for e in entries
 if e['method'] == 'POST'
 and e['url'].startswith('/checkout/')] # ✅

# Filter successful requests
successful = [e for e in entries
 if 200 <= e['status'] < 300] # ✅

# Filter mobile traffic
mobile = [e for e in entries
 if any(m in e['user_agent'].lower()
 for m in ['mobile','iphone','android','ipad'])] # ✅

# Filter redirects
redirects = [e for e in entries
 if 300 <= e['status'] < 400] # ✅

Quick Reference#

Log Format:
 IP - - [Timestamp] "Method URL Protocol" Status Size "Ref" "UA" VHost ServerIP

Valid entry needs:
 ✅ Numeric status code
 ✅ Valid HTTP method
 ✅ Valid protocol string (HTTP/1.1)

Fields for POST/path/time filter:
 ✅ Time + Method + URL

Datetime format:
 "%d/%b/%Y:%H:%M:%S %z" ✅

Weekday numbers:
 0=Mon, 1=Tue, 2=Wed, 3=Thu, 4=Fri, 5=Sat, 6=Sun

Time ranges:
 12:00-15:59 → 12 <= hour <= 15 ✅
 16:00-18:59 → 16 <= hour < 19 ✅

Request field splitting:
 split(' ')[0] → method
 split(' ')[1] → URL ✅
 split(' ')[2] → protocol ✅

URL matching:
 url.startswith('/path/') ✅

Status ranges:
 200 <= status < 300 → success ✅
 300 <= status < 400 → redirect ✅

Unexpected high count:
 → Any filter could be wrong ✅

Large file processing:
 → Generator: for line in open() ✅
 → gzip.open() for .gz files ✅

11. Data Cleaning & OpenRefine

Thu, 11 Jan 2024 00:00:00 +0000

11. Data Cleaning & OpenRefine#

1. OpenRefine - Overview#

Free, open-source tool for interactive data cleaning and reconciliation
Runs locally in browser (not cloud-based)
Works like a spreadsheet but far more powerful for messy data
❌ NOT for model training
❌ NOT for visual dashboarding
❌ NOT for source control

2. Text Clustering - Overview#

Automatically groups similar text values so you can merge them
Solves inconsistent naming problem:

"Microsoft Corp"
"MICROSOFT CORPORATION"
"microsoft"
"MSFT"
→ All same entity → cluster and merge into one

3. Clustering Algorithms#

Fingerprinting (default):#

Normalizes text → lowercase, remove punctuation, sort words, remove duplicates
Best for: simple variations in spacing/case
Example: “John Smith” and “smith john” → same fingerprint

N-gram Fingerprint:#

Uses character n-grams for comparison
Best for: slight spelling differences, abbreviations
Example: “Microsoft” and “Microsoft” (typo)

Levenshtein (Edit Distance):#

Counts minimum edits (insert/delete/replace) to transform one string to another
Best for: typos, minor errors

PPM:#

Compression-based similarity

12. Geospatial Analysis

Fri, 12 Jan 2024 00:00:00 +0000

12. Geospatial Analysis#

1. GeoPandas — Overview#

Extends pandas to support spatial/geographic data
Handles: points, lines, polygons, spatial joins, buffers, distance calculations
❌ pandas alone cannot do spatial operations
❌ matplotlib alone cannot do interactive maps

2. Creating GeoDataFrame from CSV:#

import geopandas as gpd

gdf = gpd.GeoDataFrame(
 df,
 geometry=gpd.points_from_xy(df['longitude'], df['latitude']),
 crs='EPSG:4326' # always set CRS ✅
)

3. `gpd.points_from_xy()`:#

Creates Point geometries from longitude and latitude columns
Note: longitude first, latitude second (x, y order)
❌ Common mistake: swapping lat/lon order

4. Reprojecting CRS:#

# ✅ Reproject to metric before distance calculation
gdf_metric = gdf.to_crs('EPSG:3857')

# Distance in meters now
gdf_metric.geometry.distance(city_center_point)

5. CRS — Coordinate Reference Systems#

Two Main Types:#

CRS	Code	Unit	Use For
WGS84 (Geographic)	EPSG:4326	Degrees	Storing GPS coordinates
Web Mercator (Projected)	EPSG:3857	Meters	Distance calculations ✅

The Distance Problem — Exam Scenario:#

Symptom: "Stores 1km apart show as 0.01 units apart"
Cause: Using degree-based CRS (EPSG:4326)
Fix: Reproject to metric CRS (EPSG:3857) ✅

✅ Coordinates in degree-based CRS → reproject to metric for correct distances — exam answer
❌ Data is corrupted → wrong diagnosis
❌ GeoPandas doesn’t work with lat/lon → false
❌ Multiply all distances by 100 → wrong fix

6. Distance Filtering:#

# ✅ Correct approach — exam answer (TDS Q19)
# Step 1: Create city center point
from shapely.geometry import Point
city_center = gpd.GeoDataFrame(
 geometry=[Point(77.2090, 28.6139)], # lon, lat
 crs='EPSG:4326'
).to_crs('EPSG:3857')

# Step 2: Reproject stores to metric
gdf_metric = gdf.to_crs('EPSG:3857')

# Step 3: Calculate distances in meters
gdf_metric['distance_m'] = gdf_metric.geometry.distance(
 city_center.geometry[0]
)

# Step 4: Filter stores within 5km
within_5km = gdf_metric[gdf_metric['distance_m'] < 5000]

✅ Create Point → calculate distances → filter where distance < 5000m — exam answer
❌ Manually calculate latitude difference and compare to 5 → wrong
❌ Sort by latitude and pick first 5 → wrong
❌ Use pandas string matching on store names → wrong

7. Shapely — Geometric Objects#

What is Shapely?#

Creates and manipulates geometric objects
Works alongside GeoPandas for spatial calculations

Core Objects:#

from shapely.geometry import Point, LineString, Polygon

Point(77.2090, 28.6139) # (longitude, latitude)
LineString([(0,0), (1,1), (2,0)]) # line
Polygon([(0,0),(1,0),(1,1),(0,1)])# polygon

Geometric Relationships:#

point.within(polygon) # is point inside polygon?
polygon.contains(point) # does polygon contain point?
geom1.intersects(geom2) # do they intersect?
geom1.distance(geom2) # distance between geometries
geom1.buffer(distance) # create buffer zone

8. `Point(lon, lat)` — Shapely Coordinate Order#

Shapely uses (longitude, latitude) — x, y order
❌ Common mistake: putting latitude first
Folium uses (latitude, longitude) — opposite order
Always double-check which library you’re using

9. GeoPandas Spatial Operations:#

gdf.area # area of each geometry
gdf.length # perimeter/length
gdf.centroid # center point
gdf.buffer(500) # buffer zone (500 meters if metric CRS)
gpd.sjoin(gdf1, gdf2) # spatial join

10. Folium — Interactive Maps#

What is Folium?#

Creates interactive maps rendered in web browser
Based on Leaflet.js
Output: HTML file viewable in browser
✅ Best for: stakeholder-facing interactive visualization
❌ NOT for data manipulation
❌ NOT for numerical calculations
❌ NOT for HTTP requests

11. Adding Elements to Folium Map:#

import folium

# Create base map
m = folium.Map(
 location=[28.6139, 77.2090], # [lat, lon] — note order!
 zoom_start=12
)

# Add marker
folium.Marker(
 location=[28.6289, 77.2167], # [lat, lon]
 popup='Downtown Store',
 tooltip='S001'
).add_to(m)

# Add circle (radius in meters)
folium.Circle(
 location=[28.6139, 77.2090],
 radius=5000, # 5km
 color='red',
 fill=True,
 fill_opacity=0.2
).add_to(m)

m.save('map.html')

12. Colored Markers — Coverage Analysis:#

# ✅ Exam-relevant: different colors for within/outside zone
for _, store in gdf.iterrows():
 color = 'green' if store['within_5km'] else 'red'
 folium.Marker(
 location=[store['latitude'], store['longitude']],
 popup=store['store_name'],
 icon=folium.Icon(color=color)
 ).add_to(m)

13. Saving Folium Map:#

m.save('store_map.html') # save as HTML file
# Open in any web browser — interactive ✅

14. Library Combination — Exam Answer#

Library	Role
pandas	Load CSV, data manipulation
GeoPandas	Spatial operations, distance, buffers
Shapely	Geometric object creation
Folium	Interactive map visualization

✅ GeoPandas + Shapely + Folium = most comprehensive — exam answer (May_FN Q364)
❌ Matplotlib + NumPy + Pandas → no spatial operations
❌ Only pandas → no spatial distance
❌ Only matplotlib → no interactive maps

15. Haversine Formula — Overview#

What is Haversine?#

Computes great-circle distance between two GPS points on Earth
Accounts for Earth’s spherical shape
Result in kilometers or miles

import math

def haversine(lat1, lon1, lat2, lon2):
 R = 6371 # Earth radius in km
 lat1, lon1, lat2, lon2 = map(math.radians, [lat1, lon1, lat2, lon2])
 dlat = lat2 - lat1
 dlon = lon2 - lon1
 a = math.sin(dlat/2)**2 + math.cos(lat1)*math.cos(lat2)*math.sin(dlon/2)**2
 return R * 2 * math.asin(math.sqrt(a))

dist = haversine(28.6139, 77.2090, 28.7041, 77.1025)

16. Haversine — What It Does and Doesn’t Do#

✅ Provides accurate great-circle distances for GPS coordinates — exam answer
✅ Works directly with lat/lon
✅ Does NOT require road network data
❌ Does NOT calculate exact road distances
❌ Does NOT calculate travel time
❌ Does NOT determine elevation changes
❌ Does NOT measure road surface quality

17. NetworkX — Route Optimization#

What is NetworkX?#

Python library for graph analysis and shortest paths
Models locations as nodes, roads as edges

import networkx as nx

G = nx.Graph()
G.add_edge('Hospital', 'CommunityA', weight=15.2)
G.add_edge('Hospital', 'CommunityB', weight=8.7)

# Shortest path
path = nx.shortest_path(G, 'Hospital', 'CommunityA', weight='weight')

18. OR-Tools — Vehicle Routing#

What is OR-Tools?#

Google’s library for complex optimization problems

13. Docker & Containerization

Sat, 13 Jan 2024 00:00:00 +0000

13. Docker & Containerization#

1. Docker — Overview#

What is Docker?#

Platform for packaging applications with all dependencies into containers
Ensures app runs consistently across different machines
“Build once, run anywhere”

Core Benefits:#

Benefit	Description
Consistency	Same environment on dev, staging, production
Isolation	Apps don’t interfere with each other
Portability	Run on any machine with Docker installed
Reproducibility	Exact same environment every time

❌ Does NOT make code run faster
❌ Does NOT automatically write code
❌ Does NOT provide free hosting

2. Core Docker Components:#

Docker Image
→ Blueprint/template (read-only)
→ Built from Dockerfile
→ Stored in registry (Docker Hub)

Docker Container
→ Running instance of image
→ Isolated environment
→ Has own filesystem, network

Dockerfile
→ Instructions to build image
→ Plain text file

Docker Registry
→ Docker Hub (public)
→ Private registries (ECR, GCR)

3. `docker run -p` — Port Mapping#

docker run -p 8501:8501 dashboard
# │ │
# │ └── Container port (inside Docker)
# └──────── Host port (your machine)

# Access via: http://localhost:8501

✅ docker run -p 8501:8501 dashboard — exam answer (TDS_(1) Q28)
❌ docker build -t dashboard . → builds image, doesn’t run
❌ docker push dashboard → pushes to registry, doesn’t run
❌ docker stop → stops container

4. `docker logs` — View Error Messages#

docker logs container_id # view all logs ✅
docker logs -f container_id # follow/stream logs
docker logs --tail 100 container_id # last 100 lines

✅ docker logs container_id — exam answer for debugging crashes (TDS_(1) Q29)
❌ docker build → builds image, not view logs
❌ docker stop → stops container
❌ docker remove → removes container

5. Essential Container Commands:#

docker ps # list RUNNING containers
docker ps -a # list ALL containers (including stopped)
docker stop container_id # stop running container
docker rm container_id # remove stopped container
docker rm -f container_id # force remove running container

6. Shell Access Inside Container:#

docker exec -it container_id bash # open bash shell
docker exec -it container_id sh # if bash not available
docker exec container_id env # check environment variables
docker exec container_id ls /app # check file system

7. `docker stats` — Resource Monitoring:#

docker stats # all containers
docker stats container_id # specific container

# Shows: CPU%, Memory usage, Network I/O

8. Dockerfile — Instructions Reference#

FROM python:3.11.0 # base image (use specific version ✅)
WORKDIR /app # set working directory
COPY requirements.txt . # copy dependency file first
RUN pip install -r requirements.txt # install packages ✅
COPY . . # copy application code
EXPOSE 8501 # document port (does NOT publish!)
ENV PYTHONUNBUFFERED=1 # set environment variable
CMD ["streamlit", "run", "dashboard.py"] # default run command

All Dockerfile Instructions:#

Instruction	Purpose
`FROM`	Base image
`WORKDIR`	Set working directory
`COPY`	Copy files into image
`ADD`	Copy + extract archives
`RUN`	Execute during build
`CMD`	Default run command
`ENTRYPOINT`	Fixed run command
`EXPOSE`	Document port
`ENV`	Set environment variable
`ARG`	Build-time variable
`VOLUME`	Mount point
`USER`	Set user
`HEALTHCHECK`	Health check command

9. `RUN pip install -r requirements.txt` — Exam Answer#

✅ Installs Python packages in container — exam answer (TDS_(1) Q31)
❌ COPY dashboard.py → copies file, doesn’t install
❌ EXPOSE 8501 → documents port, doesn’t install
❌ FROM python:3.11 → sets base image, doesn’t install packages

10. `EXPOSE` vs `-p` — Key Distinction#

	`EXPOSE` in Dockerfile	`-p` in `docker run`
Purpose	Documentation only	Actually publish port
Effect	None on its own	Maps host→container
Analogy	“This door exists”	“Open this door”

EXPOSE 8501 # documents port — does NOT make accessible ✅

docker run -p 8501:8501 dashboard # actually publishes port ✅

✅ EXPOSE documents the listen port; must still use -p to publish — exam answer (TDS_(1) Q32)
❌ Automatically makes app accessible → false
❌ Closes port for security → false
❌ Changes application code → false

11. Layer Caching — Best Practice:#

# ✅ Copy requirements first (changes less often)
COPY requirements.txt .
RUN pip install -r requirements.txt # cached if requirements unchanged
COPY . . # only rebuilds on code change

# ❌ Copy everything first (rebuilds pip install on every code change)
COPY . .
RUN pip install -r requirements.txt

12. Multi-Stage Builds + Service Mesh — Exam Answer#

For Distributed Edge Deployment:#

Multiple intersections/nodes → each needs container
Different services → separate containers
Resource constraints → limit CPU/memory per container

# Stage 1: Build
FROM python:3.11 AS builder
RUN pip install --user -r requirements.txt

# Stage 2: Production (smaller image)
FROM python:3.11-slim AS production
COPY --from=builder /root/.local /root/.local
COPY app.py .
CMD ["python", "app.py"]

✅ Multi-stage Docker builds with service mesh + resource constraints — exam answer (May_FN Q376)
❌ Single monolithic container → all services coupled
❌ Virtual machines → too heavy for edge devices
❌ Direct installation → no isolation

13. Resource Constraints:#

docker run \
 --cpus="0.5" \  # 50% of one CPU core
 --memory="512m" \  # 512MB RAM limit
 -p 8501:8501 \
 dashboard

14. `.dockerignore` — Exclude Files:#

.git
.env
.env.*
__pycache__
*.pyc
*.log
secrets/

15. Docker Compose — Multi-Container:#

# docker-compose.yml
version: '3.8'
services:
 dashboard:
 build: .
 ports:
 - "8501:8501"
 environment:
 - API_KEY=${API_KEY}
 depends_on:
 - database

 database:
 image: postgres:15
 environment:
 - POSTGRES_DB=analytics

docker-compose up -d # start all services
docker-compose down # stop all services
docker-compose logs # view logs

Common Docker Issues & Debugging:#

Problem	Command
App crashes on start	`docker logs container_id` ✅
Can’t access in browser	Check `-p` port mapping
Missing environment var	`docker exec container_id env`
File not found	`docker exec container_id ls /app`
Out of memory	`docker stats container_id`

Docker — Complete Command Reference:#

# Build
docker build -t name .
docker build -t name:v1.0 .

# Run
docker run -p host:container name # with port ✅
docker run -d -p 8501:8501 name # detached mode
docker run -e API_KEY=val name # with env var
docker run -v /host:/container name # with volume

# Manage containers
docker ps # running
docker ps -a # all
docker stop container_id
docker rm container_id
docker logs container_id # debug crashes ✅
docker exec -it id bash # shell access

# Manage images
docker images
docker rmi image_name
docker pull python:3.11
docker push username/app

# Cleanup
docker system prune # remove unused resources

Quick Reference#

Core concepts:
 Image → blueprint (built from Dockerfile)
 Container → running instance of image

Key commands:
 docker run -p 8501:8501 name ✅ run with port mapping
 docker logs container_id ✅ debug crashes
 docker ps list running
 docker exec -it id bash shell access

Dockerfile:
 FROM → base image
 RUN pip install -r requirements.txt ✅ install packages
 EXPOSE → document port (NOT publish!) ✅
 CMD → default run command

EXPOSE vs -p:
 EXPOSE → documentation only ✅
 -p → actually publishes port ✅

Layer caching:
 COPY requirements.txt first ✅
 THEN COPY . .

Multi-stage builds:
 → smaller final image
 → separate build and production stages ✅

Edge deployment:
 ✅ Multi-stage + service mesh + resource constraints
 ❌ Single monolithic container
 ❌ Virtual machines

14. LLMs, Prompt Engineering & RAG

Sun, 14 Jan 2024 00:00:00 +0000

14. LLMs, Prompt Engineering & RAG#

1. LLM System Prompt — Overview#

What is a System Prompt?#

Instructions given to LLM before user interaction begins
Defines model’s tone, behavior, response style, knowledge domain
Most impactful aspect = knowledge domain + expertise level ✅

Purpose:#

✅ Defines core communication strategy, learning approach, interaction guidelines — exam answer (JAN_AN Q430)
❌ Does NOT create fixed predetermined responses
❌ Does NOT prevent model from understanding context
❌ Does NOT replace human instructor guidance completely
❌ Does NOT limit AI’s ability to understand complex concepts

2. System Prompt — Knowledge Domain Impact#

What system prompt instructs LLM to adopt most impacts response
✅ Knowledge domain + expertise level — exam answer (JAN_AN Q411)
❌ Character limit set by system prompt → not most impactful
❌ Formatting requirements → secondary concern
❌ Language model version → not defined in system prompt

3. System Prompt — What It Does NOT Do#

❌ Creates fixed predetermined responses for every query
❌ Prevents deviation from pre-determined responses
❌ Configures LLM to only respond to specific commands
❌ Replaces human instructor guidance completely

4. Socratic System Prompt — Educational Use#

Best System Prompt for Educational Assistant:#

✅ "You are an interactive learning assistant for climate science.
Guide students through complex concepts by asking reflective
questions. Avoid giving direct solutions. Encourage independent
thinking and help students develop problem-solving skills."

✅ Balancing information delivery with Socratic questioning — exam answer (May_FN Q371)
❌ “Provide direct answers to maximize efficiency” → defeats educational purpose
❌ “Limiting responses to prevent information overload” → too restrictive
❌ “Using technical jargon to maintain academic rigor” → not pedagogically effective

5. Direct vs Socratic Response Style:#

Style	When
Socratic	Education, foster critical thinking ✅
Direct	Production systems, efficiency needed
Balanced	General purpose use

6. LLM POST Request — API Inference#

LLM APIs use POST requests for inference
Request body contains: model, messages, max_tokens

import requests, os

response = requests.post(
 'https://api.anthropic.com/v1/messages',
 headers={'x-api-key': os.getenv('ANTHROPIC_API_KEY')},
 json={
 'model': 'claude-3-sonnet-20240229',
 'max_tokens': 100,
 'messages': [{'role': 'user', 'content': 'Classify: Great product!'}]
 }
)
result = response.json()['content'][0]['text']

✅ POST request with review text in request body — exam answer (TDS Q28)
❌ GET request → only retrieves, can’t send body
❌ DELETE → removes resource
❌ PUT → updates existing resource

7. Prompt Specificity — Key Principle#

Vague vs Specific:#

❌ "Tell me about solar energy"
❌ "Discuss solar energy advancements"
❌ "Write a paragraph about solar energy"

✅ "Outline 4 key solar energy technological breakthroughs
 from 2014-2024, including specific efficiency improvements
 and implementation challenges"

✅ Specific prompt with: count + timeframe + aspects + constraints — exam answer
More specific = more precise and useful response

8. Specify Output Format in Prompt:#

❌ "Classify this review"
→ Output: "The sentiment is Negative."

✅ "Classify sentiment as exactly one word in lowercase:
 positive, negative, or neutral. Review: {text}"
→ Output: "negative"

9. Few-Shot Prompting:#

"Classify customer tickets:

Example 1:
Input: 'App keeps crashing'
Output: technical

Example 2:
Input: 'Wrong charge on my card'
Output: billing

Now classify:
Input: '{ticket_text}'
Output:"

10. Chain of Thought Prompting:#

"Solve this step by step:
1. First identify the main issue
2. Consider possible causes
3. Propose solution

Problem: {problem}"

11. LLM Token Costs — What Matters#

Factors that Impact Cost:#

✅ Token count of input prompts — exam answer
✅ Token count of generated responses — exam answer
✅ Query complexity requiring deeper reasoning — exam answer
✅ Context window utilization for multi-turn conversations — exam answer
❌ Time of day when queries submitted → does NOT affect cost
❌ Student’s academic level → does NOT affect cost
❌ Fixed regardless of complexity → FALSE

12. LLM Batch Processing — Production:#

import time, os
import anthropic

client = anthropic.Anthropic(api_key=os.getenv('ANTHROPIC_API_KEY'))

def analyze_reviews(reviews):
 results = []
 
 for review in reviews:
 response = client.messages.create(
 model="claude-3-sonnet-20240229",
 max_tokens=50,
 messages=[{
 "role": "user",
 "content": f"Classify as positive/negative/neutral: {review}"
 }]
 )
 results.append(response.content[0].text.strip())
 time.sleep(0.5) # rate limiting ✅
 
 return results

✅ Process in batches, track costs, handle errors, cache results — exam answer (TDS Q30)
❌ Send all 500 in single API call → context limit exceeded
❌ Only analyze 5 to save money → insufficient coverage
❌ Call API repeatedly for same review → wasteful

13. LLM Capabilities & Limitations#

What LLMs CAN Do:#

✅ Generate coherent contextual text
✅ Classify sentiment, categories
✅ Summarize documents
✅ Extract information from text
✅ Answer questions based on context
✅ Write and explain code
✅ Evaluate statistical validity when prompted well
✅ Follow formatting instructions

What LLMs CANNOT Do:#

❌ Access real-time internet (unless tool-enabled)
❌ Access live databases or patient records
❌ Cite papers published after training cutoff
❌ Guarantee factually correct information (hallucination)
❌ Guarantee consistent output format without prompting
❌ Remember previous conversations (stateless by default)

✅ LLMs may generate plausible-sounding but incorrect medical info — exam answer (JAN_AN Q412)
✅ Quality of response depends on prompt specificity — exam answer
✅ Can evaluate statistical validity when properly prompted — exam answer
❌ Can directly access post-cutoff studies → false
❌ Have real-time patient data access → false
❌ Quality unaffected by vague prompts → false

14. LLM Output Consistency — Enforcement#

Problem:#

LLM outputs:
"negative" ← correct format
"Negative" ← wrong case
"neg" ← wrong format
"NEGATIVE." ← wrong case + punctuation

Solution 1 — Prompt Engineering (Best):#

prompt = """
Respond with EXACTLY one word in lowercase: positive, negative, or neutral.
No punctuation. No explanation.
Review: {text}
"""

Solution 2 — Post-Processing (Backup):#

def normalize(output):
 output = output.strip().lower()
 if 'pos' in output: return 'positive'
 if 'neg' in output: return 'negative'
 if 'neu' in output: return 'neutral'
 return 'unknown'

✅ Use prompt engineering to enforce exact format — exam answer (TDS Q31)
❌ Manually edit each response → not scalable
❌ Ignore inconsistencies → breaks downstream analysis
❌ Switch LLMs for each review → still inconsistent

15. RAG — Overview#

What is RAG?#

Retrieval Augmented Generation
Combines retrieval of relevant documents with LLM generation
Grounds LLM in real, up-to-date knowledge
Solves: hallucination + knowledge cutoff problems

RAG vs Pure LLM:#

Aspect	Pure LLM	RAG
Knowledge	Training data only	External knowledge base
Accuracy	May hallucinate	Grounded in real docs
Updatable	Needs retraining	Update knowledge base

16. RAG Pipeline — Process Flow#

Correct Flow — Exam Answer:#

Student Query
 ↓
Vectorize Query ← convert to embedding
 ↓
Retrieve Content from ← similarity search
Vector Database
 ↓
Pass Retrieved Content ← add as context
to LLM
 ↓
LLM Generates ← context-aware response ✅
Context-Aware Response

✅ Query → Vectorize → Retrieve from vector DB → Pass to LLM → Response — exam answer (JAN_FN Q314, JAN_AN Q433)
❌ Student Query → LLM Direct Answer → Response Sent Back → skips retrieval
❌ Student Query → Chunk Course Material → Vectorize Query → wrong order
❌ Student Query → Rule-Based System → Pre-Written Response → not RAG

17. RAG Chunking — Strategy#

Why Chunk?#

LLMs have limited context windows
Smaller chunks → more precise matching
Better retrieval accuracy with focused chunks

Chunk Size — Exam Answer:#

Size	Chunks	Best For
Very small (1-2 sentences)	Too little context	❌
Medium (200-500 words)	Balance context + focus	✅ customer support
Very large (entire documents)	Too much irrelevant content	❌
Random sizes	Inconsistent retrieval	❌

✅ Medium chunks (1-2 paragraphs, ~200-500 words) — exam answer (TDS Q36)

18. RAG Stale Chunks — Exam Answer#

Problem:#

Document v1 indexed → answer from v1
Document v2 released → old chunks still in DB
→ Chatbot returns outdated information

Most Likely Cause:#

✅ Old document chunks remain in vector DB and weren’t updated — exam answer (TDS Q35)
❌ LLM relies on outdated training data → LLM uses retrieved context, not training
❌ Wrong chunking strategy → different problem
❌ Embedding model too small → different problem

19. Vector Databases#

What are They?#

Specialized databases for storing and searching embeddings
Find semantically similar content via vector similarity

Options:#

Database	Type	Best For
FAISS	Library	Local, fast prototyping
Weaviate	Full database	Production
Pinecone	Managed service	Scalable
Chroma	Open source	Simple local use

✅ FAISS / Weaviate — exam answer (JAN_FN Q314)

20. RAG — LLM Uses Chunks as Context#

LLM uses retrieved chunks as context to generate response

15. Chrome DevTools

Mon, 15 Jan 2024 00:00:00 +0000

15. Chrome DevTools#

1. Network Tab — Most Exam-Relevant#

What Network Tab Shows:#

ALL HTTP requests made by the page
Request/response headers
API call details
Load times and waterfall
Status codes
Request payload (body)
Response data

Exam Scenarios — Network Tab:#

Scenario	Answer
Examine API request headers and responses	Network tab ✅
Debug failing API call	Network tab ✅
Analyze webpage load time and resources	Network tab ✅
Identify rate limiting pattern (429 responses)	Network tab ✅
Request timing analysis + throttling simulation	Network tab ✅

Network Tab — Key Features:#

Request list → all HTTP calls
Filter by type → XHR/Fetch for API calls specifically

Click any request to see:
├── Headers tab
│ ├── Request URL
│ ├── Request Method (GET, POST)
│ ├── Status Code (200, 401, 429)
│ ├── Authorization header (where API key is)
│ └── Content-Type
│
├── Payload tab → request body (POST data)
│
├── Response tab → actual JSON response data
│
└── Timing tab → request timing breakdown
 ├── DNS lookup
 ├── Connection time
 ├── Time to First Byte (TTFB)
 └── Content download time

Throttling → simulate slow network
 (Fast 3G, Slow 3G, Custom)

Debugging with Network Tab:#

Problem: API returns 401
→ Network tab → click request → Headers tab
→ Check: is Authorization header present?
→ Check: API key format correct?

Problem: Rate limiting (429)
→ Network tab → filter XHR
→ Look for pattern of 429 responses
→ Timing tab → see request frequency
→ Fix: add time.sleep() between requests

Problem: Wrong data in response
→ Network tab → click request → Response tab
→ See exact JSON returned
→ Compare with expected structure

✅ Network tab for API debugging — exam answer (JAN_FN Q284)
✅ Network tab for load time analysis — exam answer (JAN_FN Q287)
✅ Network tab with timing analysis for rate limiting — exam answer (May_FN Q363)
❌ Console tab → JavaScript errors, not API headers
❌ Application tab → storage inspection, not API calls
❌ Performance tab → CPU/rendering bottlenecks, not API

2. Memory Tab — JavaScript Memory Leaks#

What Memory Tab Does:#

Detects JavaScript memory leaks
Heap snapshots → shows all objects in memory
Allocation timeline → tracks allocations over time
Compare snapshots → find growing object counts

Memory Leak Detection Workflow:#

1. Open DevTools → Memory tab
2. Take baseline heap snapshot
3. Reproduce the action (navigate pages, click buttons)
4. Take another heap snapshot
5. Select "Comparison" view
6. Look for:
 → Objects with positive "# New" count
 → Large "Size Delta" values

Common Causes of Memory Leaks:#

→ Event listeners not removed
→ Timers not cleared (setInterval without clearInterval)
→ DOM nodes detached but still referenced
→ Global variables holding large data
→ Closures holding references unnecessarily

✅ Memory tab for JavaScript memory leak debugging — exam answer (JAN_AN Q403)
❌ Network tab → HTTP requests, not memory
❌ Application tab → storage, not memory
❌ Console tab → JS errors, not memory profiling

3. Console Tab:#

JavaScript error messages
console.log() output
Run JavaScript commands interactively
Error filtering by level (errors, warnings, info)
❌ NOT for API header inspection
❌ NOT for memory leak detection

4. Elements Tab:#

Inspect and edit HTML and CSS in real-time
View DOM structure
Debug layout issues
Computed styles, box model visualization
❌ NOT for network requests
❌ NOT for memory profiling

5. Sources Tab:#

View JavaScript files
Set breakpoints in code
Step through code execution
Watch variables
❌ NOT for HTTP requests
❌ NOT for memory

6. Performance Tab:#

CPU usage profiling
Rendering performance
Frame rate analysis
JavaScript execution timeline
❌ NOT for API debugging
❌ NOT for memory leak detection

7. Application Tab:#

Local storage / Session storage
Cookies
IndexedDB
Service workers
Cache storage
❌ NOT for network requests
❌ NOT for memory profiling

8. Lighthouse Tab:#

Automated audits:
- Performance score
- Accessibility
- SEO
- Best practices
Provides actionable recommendations
❌ NOT for real-time debugging

9. Security Tab:#

HTTPS certificate details
Mixed content issues
Security overview

All DevTools Tabs — Complete Reference:#

Tab	Primary Use	Exam Scenario
Network	HTTP requests, API debugging, load time	API headers, 429 rate limit, load time ✅
Memory	JavaScript memory leaks	Memory leak debugging ✅
Console	JS errors, console.log output	JavaScript errors
Elements	HTML/CSS inspection and editing	Layout debugging
Sources	JS files, breakpoints, step-through	Code debugging
Performance	CPU profiling, rendering, frame rate	Performance bottlenecks
Application	Storage, cookies, service workers	Local storage inspection
Lighthouse	Automated performance/accessibility audit	Site auditing
Security	HTTPS certificate, mixed content	Security issues

DevTools — Exam Answer Mapping:#

"API request headers and responses"
→ Network tab ✅

"Webpage load time and resources"
→ Network tab ✅

"Rate limiting pattern (429)"
→ Network tab → timing analysis ✅

"JavaScript memory leak"
→ Memory tab ✅

"JavaScript errors"
→ Console tab

"Inspect/edit HTML and CSS"
→ Elements tab

"Set JavaScript breakpoints"
→ Sources tab

"CPU and rendering performance"
→ Performance tab

"Cookies and local storage"
→ Application tab

Quick Reference#

Network tab:
 ✅ API request headers + responses
 ✅ Webpage load time
 ✅ Rate limiting pattern (429)
 ✅ Request timing analysis
 ✅ Request payload (POST body)
 ✅ Response data (JSON)

Memory tab:
 ✅ JavaScript memory leak detection
 ✅ Heap snapshots
 ✅ Allocation timeline

Console tab:
 → JS errors + console.log

Elements tab:
 → HTML/CSS inspection

Sources tab:
 → Breakpoints + step-through

Performance tab:
 → CPU profiling

Application tab:
 → Storage + cookies

Lighthouse tab:
 → Automated audit scores

16. IDEs & Development Tools

Tue, 16 Jan 2024 00:00:00 +0000

16. IDEs & Development Tools#

1. IDE Built-In Features#

What is an IDE?#

Integrated Development Environment
All-in-one tool for writing, testing, debugging code
Popular for Data Science: VS Code, PyCharm, JupyterLab

Standard Built-In Features:#

Feature	Description
Syntax highlighting	Color-codes different parts of code ✅
Integrated debugger	Set breakpoints, step through code ✅
Version control integration	Git commands inside IDE ✅
Terminal access	Run commands without leaving IDE ✅
Code completion	Auto-suggests code as you type ✅
Linting	Real-time code error detection ✅
Refactoring tools	Rename variables, extract functions ✅
Search & Replace	Across entire project ✅
Extensions/Plugins	Add extra functionality ✅

2. NOT Built-In — Exam Answer#

NOT Typically Built-In:#

❌ Native hardware virtualization → needs Docker, VMware, VirtualBox
❌ Automated code review and security scanning ✅ (exam answer JAN_AN Q407)
 → needs external tools: SonarQube, Snyk, GitHub Actions

✅ Automated code review and security scanning is NOT built-in — exam answer (JAN_AN Q407)
✅ Native hardware virtualization is NOT built-in — exam answer (JAN_FN Q288)
❌ Syntax highlighting → IS built-in
❌ Integrated debugger → IS built-in
❌ Version control integration → IS built-in
❌ Terminal access → IS built-in
❌ Code completion → IS built-in

3. External Tools for Code Review:#

SonarQube → static code analysis
Snyk → security vulnerability scanning
GitHub Actions → CI/CD, automated testing
Dependabot → dependency vulnerability alerts
pre-commit → git hooks for code checks

4. Popular IDEs for Data Science:#

IDE	Best For	Key Features
VS Code	General purpose	Lightweight, huge extension library
PyCharm	Python development	Deep Python support, smart debugger
JupyterLab	Data exploration	Notebooks, inline visualization
Google Colab	Cloud notebooks	Free GPU, easy sharing
RStudio	R programming	R-specific tools
Spyder	Scientific Python	MATLAB-like interface

5. Jupyter Notebooks — Best Practices#

Structure:#

Cell 1: Imports
Cell 2: Configuration (paths, constants)
Cell 3: Load data
Cell 4: Explore data
Cell 5: Clean data
Cell 6: Analyze
Cell 7: Visualize
Cell 8: Conclusions

Before Committing to Git:#

# Clear outputs before committing
jupyter nbconvert --clear-output notebook.ipynb

# OR use nbstripout
pip install nbstripout
nbstripout notebook.ipynb

Why Clear Outputs?#

Outputs can contain sensitive data
Large outputs bloat git repository
Avoids merge conflicts in output cells
Reproducibility — others should run and get same outputs

Restart and Run All:#

Kernel → Restart & Run All
→ Ensures notebook runs top to bottom without hidden state
→ Always do this before sharing/committing

VS Code — Key Features for Data Science:#

Extensions useful for TDS:
├── Python → Python language support
├── Pylance → Fast type checking
├── Jupyter → Notebook support in VS Code
├── GitLens → Enhanced git visualization
├── Docker → Docker file support
├── REST Client → Test APIs directly in editor
└── Rainbow CSV → Color-coded CSV viewing

Debugger — How to Use:#

Setting breakpoints:
1. Click left of line number → red dot appears
2. Run in debug mode (F5)
3. Code pauses at breakpoint

Debug controls:
F5 → Continue (run to next breakpoint)
F10 → Step Over (next line, don't go into function)
F11 → Step Into (go inside function)
F12 → Step Out (exit current function)

Watch panel → monitor variable values
Call stack → see function call chain
Variables panel → all current variables

Terminal Access in IDE:#

# VS Code integrated terminal
Ctrl + ` (backtick) → open terminal
Can run:
 python script.py
 pip install pandas
 git add . && git commit -m "msg"
 pytest tests/
 docker build -t app .

uv — Python Package Manager (Exam-Relevant)#

What is `uv`?#

Fast modern Python package and project manager
Written in Rust → extremely fast (10-100x faster than pip)
Replacement for: pip + venv + pyenv combined

Correct Command Sequence — Exam Answer:#

# ✅ Correct sequence (exam answer May_FN Q359)
uv init # Step 1: initialize project
uv python install 3.11 # Step 2: install specific Python version
uv add pandas numpy # Step 3: add dependencies

# ❌ Wrong sequences:
uv init python 3.11 → uv add pandas # wrong
uv create --version 3.11 → uv install # wrong
uv new --python=3.11 → uv sync pandas # wrong

Complete `uv` Reference:#

uv init # create new project
uv python install 3.11 # install Python version
uv add pandas # add dependency
uv remove pandas # remove dependency
uv sync # install from lockfile
uv run script.py # run in project environment
uv venv # create virtual environment

`uv` vs Traditional:#

Task	Traditional	uv
Create venv	`python -m venv .venv`	`uv venv`
Install package	`pip install pandas`	`uv add pandas`
Install Python	`pyenv install 3.11`	`uv python install 3.11`
Install from requirements	`pip install -r requirements.txt`	`uv sync`

Virtual Environments — Overview:#

# venv (built-in Python)
python -m venv .venv # create
source .venv/bin/activate # activate (Mac/Linux)
.venv\Scripts\activate # activate (Windows)
deactivate # deactivate

# conda
conda create -n myenv python=3.11
conda activate myenv
conda deactivate

# uv (fastest)
uv venv
source .venv/bin/activate

Why Virtual Environments?#

Isolate project dependencies
Avoid conflicts between projects
Reproducible environments
Each project has its own Python packages

`pip` — Package Management Reference:#

pip install pandas # install
pip install pandas==2.1.0 # specific version
pip install -r requirements.txt # from file
pip uninstall pandas # remove
pip list # list installed
pip freeze > requirements.txt # save current env
pip show pandas # package info
pip install --upgrade pandas # upgrade

Quick Reference#

IDE Built-In Features:
 ✅ Syntax highlighting
 ✅ Integrated debugger
 ✅ Version control integration
 ✅ Terminal access
 ✅ Code completion
 ✅ Linting, refactoring, search

NOT Built-In:
 ❌ Native hardware virtualization ✅ (JAN_FN exam answer)
 ❌ Automated code review + security scanning ✅ (JAN_AN exam answer)

Popular IDEs:
 VS Code → general purpose, lightweight
 PyCharm → deep Python support
 JupyterLab → notebooks, data exploration

uv sequence:
 uv init → uv python install 3.11 → uv add pandas ✅

Jupyter best practices:
 ✅ Clear outputs before committing
 ✅ Restart & Run All before sharing
 ✅ Structure cells logically

Virtual environments:
 venv → built-in Python
 conda → data science focused
 uv → fastest, modern ✅

17. Kumu

Wed, 17 Jan 2024 00:00:00 +0000

17. Kumu#

1. Kumu — Overview#

What is Kumu?#

Web-based tool for creating system maps and visualizing relationships
Used for: network diagrams, stakeholder maps, concept maps, causal loop diagrams
✅ Primary use = creating system maps and visualizing relationships — exam answer (JAN_FN Q289)
❌ NOT for building financial models
❌ NOT for performing real-time data analysis
❌ NOT for designing geographic navigation maps

2. Kumu — Use Cases:#

Use Case	Description
System maps	Show interconnected components
Stakeholder maps	Relationships between people/organizations
Causal loop diagrams	Cause-effect relationships
Network analysis	Identify key nodes and connections
Impact mapping	Trace outcomes and impacts
Concept maps	Link related ideas

3. Focus Filters — Exam Answer#

What are Focus Filters?#

Highlight specific parts of visualization based on element properties

18. Python Package & Environment Management

Thu, 18 Jan 2024 00:00:00 +0000

18. Python Package & Environment Management#

1. `uv` — Overview#

Fast modern Python package and project manager
Written in Rust → 10-100x faster than pip
Replaces: pip + venv + pyenv combined
Created by Astral (makers of Ruff linter)

2. `uv` — Correct Command Sequence — Exam Answer:#

# ✅ Correct sequence (exam answer May_FN Q359)
uv init # Step 1: initialize project
uv python install 3.11 # Step 2: install specific Python version
uv add pandas numpy # Step 3: add dependencies

Wrong Sequences:#

❌ uv init python 3.11 → uv add pandas numpy
❌ uv create --version 3.11 → uv install pandas numpy
❌ uv new --python=3.11 → uv sync pandas numpy

3. `uv` — Complete Command Reference:#

# Project
uv init # create new project
uv init my-project # create in new folder

# Python versions
uv python install 3.11 # install Python 3.11
uv python list # list available versions
uv python pin 3.11 # pin version for project

# Dependencies
uv add pandas # add package
uv add pandas==2.1.0 # specific version
uv add pandas numpy scipy # multiple packages
uv remove pandas # remove package
uv sync # install from lockfile

# Running
uv run script.py # run in project environment
uv run jupyter notebook # run Jupyter

# Virtual environment
uv venv # create venv
uv venv --python 3.11 # with specific Python

# pip compatibility
uv pip install pandas
uv pip install -r requirements.txt

4. `pip` — Traditional Package Manager:#

pip install pandas # install
pip install pandas==2.1.0 # specific version
pip install -r requirements.txt # from requirements file
pip uninstall pandas # remove
pip list # list installed packages
pip freeze > requirements.txt # save current environment
pip show pandas # package details
pip install --upgrade pandas # upgrade package
pip install --no-cache-dir pandas # skip cache

5. `venv` — Built-in Virtual Environment:#

# Create
python -m venv .venv

# Activate
source .venv/bin/activate # Mac/Linux
.venv\Scripts\activate # Windows
.venv\Scripts\Activate.ps1 # Windows PowerShell

# Deactivate
deactivate

# Delete
rm -rf .venv # Mac/Linux
rmdir /s .venv # Windows

360a. `conda` — Data Science Package Manager:#

# Create environment
conda create -n myenv python=3.11

# Activate
conda activate myenv

# Deactivate
conda deactivate

# Install packages
conda install pandas numpy

# List environments
conda env list

# Export environment
conda env export > environment.yml

# Create from file
conda env create -f environment.yml

# Remove environment
conda env remove -n myenv

conda vs pip:#

Feature	pip	conda
Package source	PyPI	Anaconda + PyPI
Environment mgmt	Needs venv	Built-in
Non-Python packages	❌	✅ (C libraries etc.)
Data science focused	❌	✅
Speed	Medium	Slower

6. `requirements.txt` — Version Pinning:#

# ✅ Exact version (fully reproducible)
pandas==2.1.0
numpy==1.24.0
scikit-learn==1.3.0
matplotlib==3.7.0

# ✅ Allow patches (minor updates)
pandas>=2.0.0,<3.0.0

# ❌ Unpinned (unpredictable)
pandas
numpy

Generate requirements.txt:#

pip freeze > requirements.txt # current environment
pip install -r requirements.txt # install from file

7. `python-dotenv` — Load `.env` Files:#

pip install python-dotenv

from dotenv import load_dotenv
import os

load_dotenv() # loads .env file
api_key = os.getenv('API_KEY') # read variable
db_url = os.getenv('DATABASE_URL')

# .env file (NEVER commit to git!)
API_KEY=your_key_here
DATABASE_URL=postgresql://user:pass@localhost/db
DEBUG=false
SECRET_KEY=supersecretkey

# .gitignore — always exclude
.env
.env.*
*.env

Tool Comparison — Complete:#

Task	Traditional	uv	conda
Create venv	`python -m venv`	`uv venv`	`conda create`
Install package	`pip install`	`uv add`	`conda install`
Install Python	`pyenv install`	`uv python install`	Built-in
From requirements	`pip install -r`	`uv sync`	`conda env create`
Speed	Medium	✅ Fastest	Slowest
Data science	❌	❌	✅

Why Virtual Environments?#

Without venv:
Project A needs pandas==1.5.0
Project B needs pandas==2.1.0
→ Conflict! Can only have one globally

With venv:
Project A → its own environment → pandas==1.5.0
Project B → its own environment → pandas==2.1.0
→ No conflict ✅

Quick Reference#

uv sequence (exam answer):
 uv init → uv python install 3.11 → uv add pandas ✅

uv commands:
 uv add → add dependency
 uv remove → remove dependency
 uv sync → install from lockfile
 uv run → run in project environment

pip commands:
 pip install package==version
 pip freeze > requirements.txt
 pip install -r requirements.txt

venv:
 python -m venv .venv
 source .venv/bin/activate (Mac/Linux)
 .venv\Scripts\activate (Windows)

conda:
 conda create -n env python=3.11
 conda activate env
 conda install package

.env file:
 ✅ Store secrets here
 ✅ Load with python-dotenv
 ✅ Always add to .gitignore
 ❌ Never commit to git

requirements.txt:
 pandas==2.1.0 ✅ pinned
 pandas ❌ unpinned

19. Visualization Libraries

Fri, 19 Jan 2024 00:00:00 +0000

19. Visualization Libraries#

1. Seaborn — Overview#

What is Seaborn?#

Python library for EDA and statistical visualization
Built on top of Matplotlib
Higher-level, less code for better-looking plots
✅ Primary use = exploratory data analysis and statistical visualization — exam answer (TDS_(1) Q5)
❌ NOT for web development (Django/Flask)
❌ NOT for numerical computing (NumPy)
❌ NOT for HTTP requests (requests)

2. Seaborn — Key Plots:#

import seaborn as sns

sns.heatmap(df.corr(), annot=True) # correlation heatmap ✅
sns.histplot(df['col']) # histogram
sns.boxplot(x='cat', y='val', data=df) # box plot
sns.scatterplot(x='a', y='b', data=df) # scatter plot
sns.barplot(x='cat', y='val', data=df) # bar plot
sns.pairplot(df) # pairwise relationships
sns.violinplot(x='cat', y='val', data=df) # violin plot
sns.lineplot(x='date', y='val', data=df) # line plot

3. Seaborn — When to Use Which Plot:#

Plot	Use Case
`heatmap`	Correlation matrix visualization ✅
`histplot`	Distribution of single variable
`boxplot`	Distribution + outliers across groups
`scatterplot`	Relationship between two variables
`barplot`	Compare means across categories
`pairplot`	All pairwise relationships in dataset
`violinplot`	Distribution shape across groups
`lineplot`	Trends over time

4. Matplotlib — Overview#

What is Matplotlib?#

Low-level Python plotting library
Foundation for Seaborn and Pandas plotting
More control, more code required
Best for: custom plots, fine-tuned control

import matplotlib.pyplot as plt

plt.plot(x, y) # line plot
plt.scatter(x, y) # scatter plot
plt.bar(categories, values) # bar chart
plt.hist(data, bins=30) # histogram
plt.pie(values) # pie chart
plt.boxplot(data) # box plot

plt.xlabel('X Label')
plt.ylabel('Y Label')
plt.title('Title')
plt.legend()
plt.show()
plt.savefig('plot.png', dpi=300)

5. Matplotlib — Subplots:#

fig, axes = plt.subplots(2, 2, figsize=(12, 10))

axes[0,0].hist(data, bins=30)
axes[0,0].set_title('Histogram')

axes[0,1].boxplot(data)
axes[0,1].set_title('Box Plot')

axes[1,0].scatter(x, y)
axes[1,0].set_title('Scatter')

axes[1,1].plot(x, y)
axes[1,1].set_title('Line')

plt.tight_layout()
plt.show()

6. Plotly — Interactive Charts#

What is Plotly?#

Creates interactive charts in browser
Zoom, pan, hover tooltips built-in
Works in Jupyter notebooks and web apps

import plotly.express as px

px.line(df, x='date', y='sales', title='Sales Over Time')
px.bar(df, x='category', y='revenue')
px.scatter(df, x='advertising', y='sales', color='region')
px.histogram(df, x='age', nbins=30)
px.box(df, x='category', y='price')
px.choropleth(df, locations='country', color='value') # map

7. Folium — Interactive Maps#

Covered fully in Section 14 (Geospatial)
Creates interactive maps in browser
Best for: geographic data visualization
✅ Use for store location maps, coverage analysis

8. Streamlit — Data Dashboards#

What is Streamlit?#

Python library for building data dashboards
No HTML/CSS/JavaScript needed
Used with Docker for deployment (exam-relevant)

import streamlit as st
import pandas as pd
import plotly.express as px

st.title('Sales Dashboard')

# Sidebar filter
region = st.sidebar.selectbox('Region', df['region'].unique())
filtered = df[df['region'] == region]

# Metrics
col1, col2 = st.columns(2)
col1.metric("Total Sales", f"${filtered['sales'].sum():,.0f}")
col2.metric("Orders", len(filtered))

# Chart
fig = px.line(filtered, x='date', y='sales')
st.plotly_chart(fig)

# Table
st.dataframe(filtered)

Visualization Library Comparison — Complete:#

Library	Type	Interactive	Best For
Seaborn	Statistical	❌	EDA, statistical plots ✅
Matplotlib	General	❌	Custom, fine-tuned plots
Plotly	General	✅	Interactive charts
Folium	Maps	✅	Geographic visualization
Streamlit	Dashboard	✅	Data apps, dashboards
Bokeh	General	✅	Web-based interactive

Choosing Right Visualization — Exam Scenarios:#

"EDA and statistical visualization"
→ Seaborn ✅

"Interactive map with markers for stakeholders"
→ Folium ✅

"Deploy Python dashboard as web app"
→ Streamlit ✅

"Correlation matrix visualization"
→ sns.heatmap(df.corr()) ✅

"Coverage analysis: 8 stores within 5km"
→ Folium colored markers + radius circle ✅

"Statistical distribution + outliers"
→ sns.boxplot() ✅

"Relationship between two variables"
→ sns.scatterplot() ✅

"Trends over time"
→ px.line() or sns.lineplot() ✅

pandas Built-in Plotting:#

# Quick plots directly from DataFrame
df['col'].plot() # line plot
df['col'].plot(kind='hist') # histogram
df['col'].plot(kind='bar') # bar chart
df.plot(x='date', y='sales') # line from DataFrame
df.boxplot(column='sales', by='region') # box plot
df.hist(bins=30, figsize=(12,8)) # all columns histogram

Quick Reference#

Library Selection:
 EDA + statistical plots → Seaborn ✅
 Custom/fine-tuned plots → Matplotlib
 Interactive charts → Plotly
 Geographic/map → Folium ✅
 Data dashboards → Streamlit

Seaborn key plots:
 sns.heatmap(df.corr()) → correlation ✅
 sns.boxplot() → distribution + outliers
 sns.scatterplot() → two variable relationship
 sns.pairplot() → all pairwise relationships
 sns.histplot() → single variable distribution

Exam answers:
 "EDA and statistical visualization" → Seaborn ✅
 "Interactive map for stakeholders" → Folium ✅
 "Deploy Python dashboard" → Streamlit ✅
 "Correlation heatmap" → sns.heatmap(df.corr()) ✅

TDS Notes

1. File Formats & Data Handling

1. File Formats & Data Handling#

1. Markdown (.md)#

2. DOCX (Word Document)#

3. Binary Blob#

4. Parquet#

5. XML (eXtensible Markup Language)#

6. JSON (JavaScript Object Notation)#

7. CSV (Comma-Separated Values)#

8. File Format Comparison - Complete#

9. Excel Text-to-Columns - Auto-Date Conversion Problem#

Problem:#

Fix:#

Steps:#

10. Excel Text-to-Columns - Leading Zeros Problem#

Problem:#

Fix:#

Common cases where leading zeros matter:#

11. The Universal Fix - Format as Text First#

When to ALWAYS format as Text before import:#

In Text-to-Columns Wizard:#

12. Python csv Module#

13. Python json Module - Complete Reference#

14. Python Pickle - When NOT to Use#

Pickle Limitations:#

Quick Reference Card#

2. Shell & Command Line

2. Shell & Command Line#

1. ls - List Directory Contents#

Basic Usage:#

All Important Flags:#

Hidden Files:#

Long Format (ls -la) Output:#

2. find - Search for Files by NAME#

find vs grep:#

3. cat - Display File Contents#

4. grep - Search Text INSIDE Files#

Basic Usage:#

Important Flags:#

grep with Regex:#

5. awk - Field Extraction & Processing#

Basic Usage:#

Common Patterns:#

Special Variables:#

6. sort - Sort Output#

7. uniq - Count/Remove Duplicates#

8. head and tail - View File Portions#

9. wc - Word/Line/Character Count#

10. Shell Pipeline - Combining Commands#

The | (pipe) operator:#

Complete Log Analysis Pipeline:#

Other Useful Pipelines:#

11. dir /ah - Windows Command (NOT Unix)#

12. Other Essential Shell Commands#

Directory Navigation:#

File Operations:#

Viewing & Searching:#

Process Management:#

File Permissions:#

Network & Downloads:#

Environment Variables:#

13. Shell Scripting Basics#

Shebang and Basic Script:#

ETL Script with Idempotency:#

14. grep for Log Analysis - Exam Scenarios#

Scenario 1: Find entries related to specific term#

Scenario 2: Search across multiple files#

Scenario 3: Extract specific log entries#

Shell Commands - Quick Reference Card#

3. Python File & Memory Handling

3. Python File & Memory Handling#

1. Generator Expression - Memory-Efficient File Processing#

What is a Generator?#

The Problem with Loading Entire File:#

Generator Solution:#

Generator Expression vs List Comprehension:#

Generator with yield:#

Practical Example - Large Log Processing:#

2. readlines() - Why Avoid for Large Files#

1. Markdown (`.md`)#

12. Python `csv` Module#

13. Python `json` Module - Complete Reference#

1. `ls` - List Directory Contents#

Long Format (`ls -la`) Output:#

2. `find` - Search for Files by NAME#

`find` vs `grep`:#

3. `cat` - Display File Contents#

4. `grep` - Search Text INSIDE Files#

5. `awk` - Field Extraction & Processing#

6. `sort` - Sort Output#

7. `uniq` - Count/Remove Duplicates#

8. `head` and `tail` - View File Portions#

9. `wc` - Word/Line/Character Count#

The `|` (pipe) operator:#

11. `dir /ah` - Windows Command (NOT Unix)#

14. `grep` for Log Analysis - Exam Scenarios#

Generator with `yield`:#

2. `readlines()` - Why Avoid for Large Files#

3. `pd.read_csv(chunksize=N)` - Large CSV Processing#

Options with `read_csv`:#

4. `gzip.open()` - Reading Compressed Files#

5. `with open()` - Context Manager#

6. `os.path` - File Path Operations#

7. `pathlib.Path` - Modern File Path Handling#

8. `tempfile` - Temporary Files#

9. `shutil` - File Operations#

Why `shutil.move()` is Atomic:#

3. `.gitignore` - Excluding Files from Tracking#

What is `.gitignore`?#

Common `.gitignore` Entries:#

`.gitignore` vs `.git/config`:#

4. `git diff` - View Changes#

`git diff` Output:#