<
General Instructions for the ML Coding Problems¶
Please follow these instructions carefully to ensure a smooth evaluation process.
1. Notebook Submission¶
- You must make a copy of this notebook and append your full name to the filename before submitting (e.g.,
[OriginalNotebookName]_[YourName].ipynb). - Share your notebook copy with inaio@acmindia.org [This is for your own safety so that you do not accidentally lose any changes while editing the notebook]
- After solving the questions, ensure you mention the correct URL of your modified notebook in the test form
- Also answer questions on external resources used and link to LLM chats used for each problem in the main test form
2. Attempting the Questions¶
- Carefully read each problem statement before attempting.
- Attempt all parts of each question.
- Each question is organized into the following parts
- DATA, TASK, HELPER CODE [Optional] and ANSWER
- Follow the function signatures provided. Do not modify them.
- You only need to edit the cells in the ANSWER sections
- If required, you may also add other modules under IMPORTS and INSTALLATION INSTRUCTIONS
- Do not edit the other cells, especially those marked with DO NOT MODIFY which are meant for evaluation
- You may add new cells to the notebook with extra code as desired
3. Scoring Criteria¶
Your score will be based on the following factors with distribution varying across each problem.
- Soundness & Creativity of your approach.
- Include a clear description and rationale of your solution methodology in the notebook (in markdown cells)
- Solutions that showcase your understanding of data and ML will garner more points
- Code Implementation & Readability
- Ensure your implementation is correct and works
- Incomplete non-working code will be awarded partial marks based on problem-wise rubric
- In case you have a solution but are unsure about some aspect, you can define a function that solves that aspect and present the rest of the solution
- Use comments to explain important parts of your code.
- Performance of Your Model:
- Each task will be assessed based on specified performance metrics both on shared datasets and secret datasets
- Different performance ranges will receive different scores.
- Secret datasets used for last section will be shared along with the final results
Points associated with cells are marked at the beginning of the cell
4. Dataset Usage¶
- Only use the datasets provided in this test.
- Do not use the provided test data set for training.
- Do not use external datasets for training or testing.
- If the submitted performance metrics cannot be reproduced with your code and original datasets, then you will lose all the points associated with model performance.
In [ ]:
Problem 6: Drone Patrol: From Vivid Landscapes to Monochrome Missions [14 pts]¶
You are an AI military expert assigned to modernize army surveillance by setting up an unmanned AI drone patrol system. The drones are equipped with cameras and microphones to detect unusual activity. During patrol, whenever an unusual sound is detected, the drone captures an image of the source location. The challenge is to classify the source location to determine if it is on the coast or desert and inform the appropriate responding authority.
Training Data:
- We have a large corpus of previous images that can be used for training
- During the day, the drone captures color images, and these images are all labeled by human experts.
- At night, the drone captures only grayscale images, but these remain unlabeled since it takes more effort.
During patrol, the new images to be classified can be colored or grayscale based on the whether it is daytime or night.
Your task is to develop an ML model that can learn from the training data (labeled color images and the unlabeled grayscale images) to perform accurate classification during patrol.
This problem has 3 questions (2 to be attempted, 3rd one private INAIO evaluation)
- Q1: Build an Image LocationType Classifier [9 pts]
- Q2: Test Your Classifier - Public Set [2 pts]
- Q3: Test Your Classifier - Secret Set [3 pts] [NOT FOR STUDENTS TO ATTEMPT]
INSTALLATION¶
In [ ]:
!pip install uv
!uv pip install tensorflow pandas numpy opencv-python-headless scikit-learn matplotlib seaborn
IMPORTS¶
In [ ]:
# EDIT:
# You may add any other free python packages along with comments
# Data Types
from typing import Any
# System, File I/O
import os
import cv2
# Data handling
import pandas as pd # Data manipulation and analysis
import numpy as np # Numerical computations and array handling
# Machine Learning models and process
import tensorflow as tf
from tensorflow.keras.preprocessing.image import load_img, img_to_array
from sklearn.model_selection import train_test_split # Split data into training and testing sets
from sklearn.preprocessing import LabelEncoder
# Model evaluation
from sklearn.metrics import accuracy_score, classification_report # Performance metrics
from sklearn.metrics import roc_curve, auc # ROC curve and AUC score
# Visualization
import matplotlib.pyplot as plt # Plotting graphs
import seaborn as sns # Enhanced data visualization
# Image processing
from skimage.color import rgb2gray
from skimage.io import imread
from skimage.metrics import structural_similarity as ssim
COPY DATA
In [ ]:
# Copy data
!mkdir /content/data
!wget https://raw.githubusercontent.com/inaiogit/stage2test/main/test/color_images_test_public.csv
!wget https://raw.githubusercontent.com/inaiogit/stage2test/main/test/color_images_train.csv
!wget https://raw.githubusercontent.com/inaiogit/stage2test/main/test/grayscale_images_test_public.csv
!wget https://raw.githubusercontent.com/inaiogit/stage2test/main/test/grayscale_images_train.csv
!wget https://raw.githubusercontent.com/inaiogit/stage2test/main/test/processed_images.zip
!mv color_images_test_public.csv color_images_train.csv grayscale_images_test_public.csv grayscale_images_train.csv data/
!unzip processed_images.zip
!rm processed_images.zip
Q1: Build an Image Location Type Classifier [9 pts]¶
DATA¶
You are provided with two datasets:
- color_images_train: one image per row (image_path, label:"Coast" vs. "Desert")
- grayscale_images_train: one image per row (image_path)
- images/ : folder with image files
In [ ]:
# Training datasets
color_train_path = "data/color_images_train.csv" # Labeled colored images - two columns image_path, label
grayscale_train_path = "data/grayscale_images_train.csv" # Unlabeled grayscale images - single column image_path
TASK¶
Create two functions learn_location_type_classifier and classify_image as per the signatures defined below: If you scroll down, you will see cells with the skeletal code that you need to flesh out.
Function 1:learn_location_type_classifier¶
def learn_location_type_classifier(color_train_path: str, grayscale_train_path: str) -> Any:
"""
Train a classifier to distinguish between Coast and Desert images.
Parameters:
- color_train_path (str): csv file containing file path and label ofcolor images.
- grayscale_train_path (str): csv file containing file paths of unlabeled grayscale images.
Returns:
- model (Any): Trained classification model that includes preprocessing.
"""
passFunction 2: classify_image¶
def classify_image(image_path: str, model: Any) -> str:
"""
Classify an image as either "Coast" or "Desert".
Parameters:
- image_path (str): Path to the image file.
- model (Any): Trained classification model that includes preprocessing.
Returns:
- label (str): Predicted label, either "Coast" or "Desert".
- float: A probability score where:
- values close to 1 indicates high confidence the image is "Coast".
- values close to 0 indicates high confidence the image is "Desert".
"""
passHints:
- Load and explore the data
- Identify potentially discriminating features
- Come with a strategy for building a classifier
- Train models, test, and iterate
HELPER CODE¶
In [ ]:
# HELPER CODE
# You may choose to use or modify any of the below code in your solution, but it is NOT mandatory
def convert_to_grayscale(image_path: str, output_dir: str = None) -> str:
"""
Converts an image to grayscale and saves it with 'grayscale_' prefixed to the filename.
Parameters:
- image_path (str): Path to the original image file.
- output_dir (str, optional): Directory where the grayscale image will be saved.
If None, saves in the same directory as the input image.
Returns:
- str: Path to the saved grayscale image.
"""
# Read the image
image = cv2.imread(image_path)
if image is None:
raise ValueError(f"Could not read image: {image_path}")
# Convert to grayscale
grayscale_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
# Construct output file name with prefix
base_name, ext = os.path.splitext(os.path.basename(image_path))
grayscale_filename = f"grayscale_{base_name}{ext}"
# Determine output directory
if output_dir is None:
output_dir = os.path.dirname(image_path)
os.makedirs(output_dir, exist_ok=True)
# Save grayscale image
output_path = os.path.join(output_dir, grayscale_filename)
cv2.imwrite(output_path, grayscale_image)
print(f"Grayscale image saved to: {output_path}")
return output_path
def train_image_classifier(df: pd.DataFrame, epochs: int = 10, test_size: float = 0.2) -> Any:
IMG_SIZE = (128, 128)
BATCH_SIZE = 32
def load_images_labels(df):
images = []
labels = []
for _, row in df.iterrows():
img_path = row["image_path"]
label = row["label"]
if os.path.exists(img_path):
image = load_img(img_path, target_size=IMG_SIZE)
image = img_to_array(image) / 255.0
images.append(image)
labels.append(label)
else:
print(f"Warning: {img_path} does not exist.")
return np.array(images), np.array(labels)
images, labels = load_images_labels(df)
# Check if any images were loaded
if len(images) == 0:
raise ValueError("No images loaded. Please verify your CSV file and image paths.")
label_encoder = LabelEncoder()
labels = label_encoder.fit_transform(labels)
X_train, X_val, y_train, y_val = train_test_split(
images, labels, test_size=test_size, random_state=42
)
model = tf.keras.models.Sequential([
tf.keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(IMG_SIZE[0], IMG_SIZE[1], 3)),
tf.keras.layers.MaxPooling2D(2, 2),
tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
tf.keras.layers.MaxPooling2D(2, 2),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.Dense(1, activation='sigmoid')
])
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=epochs, batch_size=BATCH_SIZE)
test_loss, test_acc = model.evaluate(X_val, y_val)
print(f"Validation Accuracy: {test_acc:.4f}")
return model, label_encoder
def classify_image(image_path: str, model: tf.keras.Model) -> float:
"""
Classifies a new image and returns a probability score (0 to 1).
Parameters:
- image_path (str): Path to the image to be classified.
- model (tf.keras.Model): Trained CNN model.
Returns:
- float: Probability score (0 to 1), where closer to 1 means one class and closer to 0 means the other.
"""
IMG_SIZE = (128, 128)
if not os.path.exists(image_path):
raise ValueError(f"Image path does not exist: {image_path}")
image = load_img(image_path, target_size=IMG_SIZE)
image = img_to_array(image) / 255.0
image = np.expand_dims(image, axis=0)
score = model.predict(image)[0][0]
return score
def compute_image_similarity(image_path1: str, image_path2: str) -> dict:
"""
Computes similarity between two images using SSIM and MSE.
Parameters:
- image_path1 (str): Path to the first image.
- image_path2 (str): Path to the second image.
Returns:
- dict: Dictionary containing SSIM and MSE scores.
"""
# Load the images
image1 = cv2.imread(image_path1, cv2.IMREAD_GRAYSCALE)
image2 = cv2.imread(image_path2, cv2.IMREAD_GRAYSCALE)
if image1 is None or image2 is None:
raise ValueError("One or both image paths are invalid.")
# Resize images to the same size
image1 = cv2.resize(image1, (128, 128))
image2 = cv2.resize(image2, (128, 128))
# Compute SSIM (Structural Similarity Index)
ssim_score = ssim(image1, image2)
# Compute MSE (Mean Squared Error)
mse_score = np.mean((image1 - image2) ** 2)
return {"SSIM": ssim_score, "MSE": mse_score}
ANSWER¶
EDIT: [4 pts]¶
You can jot initial notes here and flesh this out in more detail after the implementation.¶
Describe Your Solution Approach¶
• Data Exploration Approach [0.5 pt]¶
- Review the provided CSV files to verify that image paths for both labeled (color) and unlabeled (grayscale) images are correct.
- Visualize a few samples from each set to understand image quality and class distinctions.
• Instance Representation/Preprocessing [0.5 pt]¶
- Resize all images to a consistent size (128×128) and normalize pixel values to the range [0, 1].
- Maintain color images as is, while converting grayscale images as needed using helper functions.
• Modeling Strategy [2 pt]¶
- Train an initial CNN classifier using the 50 labeled color images.
- Apply a self-training strategy: use the initial model to pseudo-label the 150 unlabeled grayscale images when predictions are highly confident.
- Combine the pseudo-labeled images with the original labeled dataset and retrain the CNN model to improve overall performance.
• Specific Modeling Choices [1 pt]¶
- Use a simple CNN architecture with two convolutional layers followed by max-pooling, a flattening layer, dense layers, and dropout for regularization.
- Employ a sigmoid activation in the final layer and binary cross-entropy as the loss function.
- Set high confidence thresholds (e.g., ≥0.9 for "Coast" and ≤0.1 for "Desert") to ensure reliable pseudo-labeling.
In [ ]:
# EDIT: [0.5 pt]
# Add your data exploration code here
color_data = pd.read_csv(color_train_path)
color_data.head()
Out[ ]:
In [ ]:
color_data.label.unique()
Out[ ]:
In [ ]:
color_data.info()
In [ ]:
grayscale_data = pd.read_csv(grayscale_train_path)
grayscale_data.head()
Out[ ]:
In [ ]:
grayscale_data.info()
In [ ]:
# EDIT: [1 pt]
# Add any instance representation/preprocessing code here
In [ ]:
# EDIT: [O pts]
# Add any additional code that you need for your classifier implementation
# Code in this cell will be evaluated with points assigned to the classifier training and prediction implementation cells
In [ ]:
# EDIT: [2.5 pts]
# Implement the classifier training
# You can choose to edit the partial implementation but keep the signature same
def learn_location_type_classifier(color_train_path: str, grayscale_train_path: str) -> dict:
"""
Train a classifier to distinguish between Coast and Desert images.
We first train on the 50 labeled color images. Then we use the trained model to
pseudo–label the 150 unlabeled grayscale images when the prediction confidence is high.
We then retrain the model on the combined dataset.
Parameters:
- color_train_path (str): CSV file containing file path and label of color images.
(CSV is assumed to have columns: "image_path" and "label")
- grayscale_train_path (str): CSV file containing file paths of unlabeled grayscale images.
(CSV is assumed to have a column: "image_path")
Returns:
- dict: Contains the final trained model and the label encoder, e.g.,
{"model": cnn_model, "label_encoder": label_encoder}
"""
# Load CSV files
df_color = pd.read_csv(color_train_path)
df_gray = pd.read_csv(grayscale_train_path)
# Train an initial classifier on the labeled (color) images.
print("Training initial model on labeled color images...")
cnn_model, label_encoder = train_image_classifier(df_color, epochs=10)
# Self-training: pseudo-label the grayscale images
pseudo_data = []
high_conf_threshold = 0.9 # high confidence for "Coast"
low_conf_threshold = 0.1 # high confidence for "Desert"
print("Pseudo-labeling unlabeled grayscale images...")
for idx, row in df_gray.iterrows():
image_path = row["image_path"]
try:
prob = _predict_probability(image_path, cnn_model)
except Exception as e:
print(f"Skipping {image_path} due to error: {e}")
continue
# If the model is very confident, assign a pseudo-label
if prob >= high_conf_threshold:
pseudo_label = "Coast"
pseudo_data.append({"image_path": image_path, "label": pseudo_label})
elif prob <= low_conf_threshold:
pseudo_label = "Desert"
pseudo_data.append({"image_path": image_path, "label": pseudo_label})
# Otherwise, do not add the image to the training set.
if pseudo_data:
df_pseudo = pd.DataFrame(pseudo_data)
df_combined = pd.concat([df_color, df_pseudo], ignore_index=True)
print(f"Retraining with {len(df_pseudo)} pseudo-labeled grayscale images (total training samples: {len(df_combined)})...")
cnn_model, label_encoder = train_image_classifier(df_combined, epochs=10)
else:
print("No pseudo-labeled grayscale images met the confidence threshold.")
# Return a dictionary with the model and label encoder
return {"model": cnn_model, "label_encoder": label_encoder}
In [ ]:
# EDIT: [1 pt]
# Implement the classifier prediction function
# You can choose to edit the partial implementation but keep the signature same
def classify_image(image_path: str, classifier: dict) -> (str, float):
"""
Classify an image as either "Coast" or "Desert".
This function loads the image, preprocesses it, and uses the trained CNN model to predict
a probability score. If the probability is >= 0.5, the image is labeled as "Coast";
otherwise it is labeled as "Desert".
Parameters:
- image_path (str): Path to the image file.
- classifier (dict): A dictionary containing the trained model and label encoder,
as returned by learn_location_type_classifier.
Returns:
- label (str): Predicted label, either "Coast" or "Desert".
- prob (float): Probability score where values close to 1 indicate "Coast" and values
close to 0 indicate "Desert".
"""
IMG_SIZE = (128, 128)
cnn_model = classifier["model"]
# Here we assume that load_img loads the image in RGB mode even if it is grayscale.
image = load_img(image_path, target_size=IMG_SIZE)
image = img_to_array(image) / 255.0
image = np.expand_dims(image, axis=0)
prob = cnn_model.predict(image)[0][0]
label = "Coast" if prob >= 0.5 else "Desert"
return label, prob
Q2: Test Your Classifier - Public Set [2 pts]¶
DATA¶
We now have datasets for patrol time classification
Use this as your test set. Do not use this for training or validation!
- color_test_public_path: Labeled color images (one image per row, two columns - image path, label)
- grayscale_test_public_path: Labeled grayscale images (one image per row, two columns - image path, label)
In [ ]:
# Public Test datasets
color_test_public_path ="data/color_images_test_public.csv" # Labeled color images (one image per row, two columns - image path, label)
grayscale_test_public_path="data/grayscale_images_test_public.csv" # Labeled grayscale images [one image per row, two columns - image path, label
TASK¶
Execute the code below as is with your implementation of learn_location_type_classifier and classify_image to test your classifier
- Evaluate your model on the two public test splits (color_test_public and grayscale_test_public)
- Generate the ROC (Receiver Operating Characteristic) curve to assess your classifier's performance.
- Compute the AUC (Area Under Curve) value as a performance metric.
HELPER CODE¶
In [ ]:
# DO NOT MODIFY
# Use these functions directly since these are meant for evaluation
def plot_roc_curve(y_true, y_scores):
fpr, tpr, _ = roc_curve(y_true, y_scores)
roc_auc = auc(fpr, tpr)
plt.figure(figsize=(8, 6))
plt.plot(fpr, tpr, color='blue', lw=2, label=f'ROC curve (AUC = {roc_auc:.2f})')
plt.plot([0, 1], [0, 1], color='gray', linestyle='--') # Random classifier line
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve')
plt.legend(loc='lower right')
plt.show()
print(f"AUC Value: {roc_auc:.4f}")
return roc_auc
def evaluate_classifier(dataset_path: str, model) -> float:
"""
Loads a labeled dataset, applies the classify_poem function with the given model,
and computes the ROC curve & AUC.
Parameters:
- dataset_path (str): Path to the CSV dataset containing 'poem' and 'label' columns.
- model (Any): A trained model that will be used by classify_poem.
Outputs:
- Plots the ROC curve
- Prints the AUC value
Returns:
- float: The AUC (Area Under the Curve) value.
"""
# Load the dataset
df = pd.read_csv(dataset_path)
# Ensure required columns exist
if "image_path" not in df.columns or "label" not in df.columns:
raise ValueError("Dataset must contain 'image_path' and 'label' columns")
# Apply the classifier to get prediction scores
df["predicted_score"] = df["image_path"].apply(lambda x: classify_image(x,model))
# Extract true labels and predicted scores
y_true = df["label"].values # Ground truth (1 = A, 0 = B)
y_scores = df["predicted_score"].values # Model output scores
# Compute ROC curve and AUC
plot_roc_curve(y_true, y_scores)
return roc_auc
In [ ]:
# DO NOT MODIFY
# Run this code and observe the ROC Curve and AUC [pts depend on performance range]
model = learn_location_type_classifier(color_train_path,grayscale_train_path)
auc_color_test_public=evaluate_classifier(color_test_public_path, model)
auc_grayscale_test_public=evaluate_classifier(grayscale_test_public_path, model)
print(auc_color_test_public)
print(auc_grayscale_test_public)
ANSWER¶
YOU CAN STOP THE TEST HERE -- BELOW EVALUATION TO BE PERFORMED BY INAIO¶
Q3: Test Your Classifier - Secret Set [3 pts]¶
- Same metrics as Public Dataset
In [ ]: