Create README.md

multimodal-bnn/README.md

@@ -0,0 +1,106 @@
+---
+license: mit
+language:
+- en
+metrics:
+- accuracy
+base_model:
+- microsoft/resnet-50
+pipeline_tag: image-classification
+tags:
+- AUV
+- biology
+- marine
+- sonar
+- Image
+- Multimodal
+- ecology
+- robotics
+---
+
+# Model Card: Multimodal Bayesian Neural Network Classifier
+
+This model is a Bayesian MultiModal Neural Network (BNN) classifier designed for the classification of AUV (Autonomous Underwater Vehicle) sensor data. It integrates information from three modalities: image, bathymetry, and side-scan sonar (SSS) data. The model utilizes custom ResNet50-based feature extractors for each modality, followed by additive attention mechanisms to fuse the features before final classification. As a BNN, it provides uncertainty estimates alongside its predictions, which can be valuable for decision-making in uncertain underwater environments.
+
+---
+
+## Model Details
+
+### Model Description
+
+This model is a Bayesian MultiModal Neural Network (BNN) classifier, specifically the `MultiModalModel` implementation. It is designed for the classification of AUV sensor data by integrating information from three distinct modalities: image, bathymetry, and side-scan sonar (SSS) data.
+
+The architecture comprises:
+* **Three `ResNet50Custom` feature extractors**: One for each modality (image, bathymetry, SSS). These are based on a `resnet50` backbone pre-trained on ImageNet-1K, adapted for specific input channel requirements (3 for image and bathymetry, 1 for SSS).
+* **Additive Attention Mechanisms**: Features extracted from each modality are then passed through separate `AdditiveAttention` layers to fuse the information effectively.
+* **Fully Connected Layers**: The concatenated, attended features are fed into a series of fully connected layers for final classification into 7 distinct categories.
+
+As a **Bayesian Neural Network**, this model provides uncertainty estimates alongside its predictions, which is a crucial capability for robust decision-making in complex and uncertain underwater environments.
+
+* **Developed by:** Sams-Tom
+* **Shared by:** Sams-Tom
+* **Model type:** Bayesian Neural Network (BNN), Multi-modal Classifier (Computer Vision / Sensor Fusion)
+* **Language(s) (NLP):** N/A (Not an NLP model)
+* **License:** MIT
+* **Finetuned from model [optional]:** ResNet50 (pre-trained on ImageNet-1K for feature extraction)
+
+### Model Sources
+
+* **Repository:** `https://huggingface.co/sams-tom/multimodal-auv-bathy-bnn-classifier`
+* **Paper:** [In development]
+* **Demo [optional]:** [More Information Needed: If there's a live demo or notebook, please link it here.]
+
+
+---
+
+## How to Get Started with the Model
+
+To use this model, ensure you have `torch`, `huggingface_hub`, and `bayesian-torch` installed (`pip install torch huggingface_hub bayesian-torch`).
+
+```python
+from transformers import AutoModel
+import torch
+import json
+import os
+from huggingface_hub import hf_hub_download
+
+# Assuming your custom model definitions (Identity, AdditiveAttention, ResNet50Custom, MultiModalModel)
+# are uploaded as model_definitions.py in the root of your repo.
+# If they are not, you will need to define them locally or adjust the import path.
+
+# Define the repository ID and the subfolder where this specific model is located
+repo_id = "sams-tom/multimodal-auv-bathy-bnn-classifier"
+model_subfolder = "multimodal-bnn"
+
+# Load the BNN prior parameters that were used during training/conversion.
+# These are essential for understanding the model's Bayesian properties.
+bnn_params_path = hf_hub_download(repo_id=repo_id, filename=os.path.join(model_subfolder, "bnn_params.json"))
+with open(bnn_params_path, "r") as f:
+    const_bnn_prior_parameters = json.load(f)
+print(f"Loaded BNN Prior Parameters: {const_bnn_prior_parameters}")
+
+# Load the model directly using AutoModel, specifying the subfolder.
+# trust_remote_code=True is crucial because your model uses custom classes (MultiModalModel, ResNet50Custom, etc.)
+# which are defined in the 'model_definitions.py' file uploaded to the root of the repository.
+model = AutoModel.from_pretrained(repo_id, subfolder=model_subfolder, trust_remote_code=True)
+model.eval() # Set to evaluation mode
+
+# Example inference with dummy data
+# Adjust dimensions (batch size, channels, height, width) based on your actual data
+dummy_image_input = torch.randn(1, 3, 224, 224) # Example: (Batch, RGB Channels, Height, Width)
+dummy_bathy_input = torch.randn(1, 3, 224, 224) # Example: (Batch, RGB Channels, Height, Width)
+dummy_sss_input = torch.randn(1, 1, 224, 224)   # Example: (Batch, Gray Channels, Height, Width)
+
+with torch.no_grad():
+    outputs = model(dummy_image_input, dummy_bathy_input, dummy_sss_input)
+    probabilities = torch.softmax(outputs, dim=-1) # Get class probabilities
+    predicted_class = torch.argmax(probabilities, dim=-1) # Get predicted class index
+
+print(f"Output logits: {outputs}")
+print(f"Predicted class probabilities: {probabilities}")
+print(f"Predicted class index: {predicted_class.item()}")
+
+# You can also inspect the model's Bayesian layers if needed
+# from bayesian_torch.models.dnn_to_bnn import dnn_to_bnn_params
+# bnn_layers_info = dnn_to_bnn_params(model)
+# print("Bayesian layers info:", bnn_layers_info)