Immich.app and Nextcloud integration for a perfect media viewing and face recognition/classification experience

Integrating Immich.app with Nextcloud as a Media Server: A Superior Solution for Facial Recognition and Media Management. Faster and more stable than Nextcloud Recognize and the abandoned Facerecognition app. In comparison:

Here is the comparison in table form:

Calculated Values:

1. Nextcloud Recognize Face Detection Rate:
   • Recognized images: 3,122
   • Total images (using Immich as reference): 12,233
   • Percentage recognized: 3,12212,233×100≈25.5%\frac{3,122}{12,233} \times 100 \approx 25.5 \%12,2333,122​×100≈25.5%
2. Nextcloud Facerecognition Face Detection Rate:
   • Recognized images: 9,220
   • Total images (using Immich as reference): 12,233
   • Percentage recognized: 9,22012,233×100≈75.4%\frac{9,220}{12,233} \times 100 \approx 75.4 \%12,2339,220​×100≈75.4%

Pros/Cons:

• Pros of Immich:
• High Detection Accuracy: Immich boasts an impressive 99.8% face detection rate, making it highly reliable for identifying faces in images and videos.
• GPU Acceleration: Supports CUDA 12.6 for machine learning tasks, making face detection, recognition, and object recognition faster by utilizing GPU resources. Also comes with vaapi and other GPU support.
• Hardware Video Transcoding: Immich allows hardware-accelerated transcoding using NVIDIA GPUs, significantly improving performance for video handling, especially for large media libraries.
• Video Face Detection: One of the standout features is the ability to detect faces in videos, with 1,728 faces detected in videos, which neither Nextcloud Recognize nor Nextcloud Facerecognition support.
• Wide Format Support: Immich supports a wide range of formats for both images and videos, including mp4, jpg, tif, png, Sony ARW, Fuji RAW, Canon RAW, covering most use cases for professional photographers and media enthusiasts.
• Efficient Performance: The total face and object detection for the entire library takes just 49.5 hours, which includes machine learning tasks like object recognition.
• Automatic Object Detection: Immich provides object recognition, adding another layer of media organization and searchability that can be very useful for users.
• Better Cluster Management: Immich has better management of face clusters, reducing manual effort in reassigning or organizing faces as compared to Nextcloud Facerecognition.
• Multi-device Synchronization: Immich is designed with synchronization in mind, allowing seamless media organization and machine learning tasks across multiple devices.
• Modern, User-friendly Interface: The Immich app offers a clean, responsive, and modern user interface, making it easy to navigate and manage large volumes of media.

Cons of Nextcloud Recognize:

• Low Face Detection Rate: Nextcloud Recognize struggles with a low face detection rate, recognizing only 0-3,122 faces, which is about 25.5% of the faces detected by Immich.
• No Video Face Detection: Nextcloud Recognize does not support face detection in videos, making it less versatile for users who work with video content.
• CUDA Incompatibility: While it supports CUDA 11.2 and PyTorch, it’s not usable in certain configurations, such as when a system uses newer versions (e.g., CUDA 12.6). This means users might not benefit from GPU acceleration if they can’t match the exact CUDA version.
• Long Processing Time: The face detection and object recognition process takes 296.5 hours for the entire library, which is significantly slower compared to Immich, mainly because of the lack of effective GPU utilization and optimized algorithms.
• Limited Face Recognition Accuracy: A small percentage of your face is detected, leading to inconsistent and incomplete recognition results, making the solution less effective for comprehensive face tagging.
• Lack of Object Recognition: The object recognition feature, while present, does not function as efficiently as it does in Immich, especially when combined with the already slow processing times.
• Poor Raw Image Handling: While it supports many formats, the performance when handling large or RAW image files is subpar compared to Immich.

Cons of Nextcloud Facerecognition:

• Manual Cluster Management: Nextcloud Facerecognition has poor face cluster management, requiring users to manually reassign faces every time the system opens a new cluster, which can be highly time-consuming and inefficient.
• No Video Support: Like Nextcloud Recognize, Nextcloud Facerecognition does not support face detection in videos, leaving out an important media type for many users.
• Moderate Detection Rate: Although it performs better than Nextcloud Recognize, Nextcloud Facerecognition detects only 9,220 faces, which is 75.4% of what Immich detects. This means around 24.6% of faces are missed.
• No Object Recognition: Unlike Immich, Nextcloud Facerecognition does not offer object detection, reducing its overall utility for users looking to organize their media beyond just faces.
• Slower Performance: Face detection in the entire library takes 102 hours, which, while faster than Nextcloud Recognize, is still over double the time Immich requires.
• Lack of Modern GPU Support: Nextcloud Facerecognition only works with dlib and pdlib for GPU support, which can limit its performance compared to more modern machine learning frameworks used by Immich.
• Cluster Duplication Issues: Clusters of recognized faces often get duplicated or reopened unnecessarily, leading to a confusing user experience and requiring extra manual effort to clean up the dataset.
• Difficult to Scale: As media libraries grow larger, Nextcloud Facerecognition becomes harder to scale efficiently due to performance and management issues.
• Outdated Algorithms: The face detection algorithms are less advanced compared to those used in Immich, contributing to lower accuracy and slower processing times.
• These detailed pros and cons highlight why Immich is currently more advanced in terms of speed, accuracy, and functionality for face recognition and media handling, particularly for users with large image and video libraries who want efficient performance and ease of use.

Nextcloud has become a popular choice for individuals and organizations looking for a self-hosted cloud solution. Among its many features, Nextcloud offers several options for managing and viewing photos, including the Photos app and the Memories app. These apps also provide some basic facial recognition capabilities through integrations with tools like Recognize and Facedetection. However, these solutions have significant limitations, which often leave users frustrated with their performance and accuracy. This article explores how integrating Immich.app into Nextcloud as a media server can offer a far superior experience, thanks to its better face detection and classification capabilities, as well as advanced features like hardware acceleration.

Limitations of Current Nextcloud Photo Management Solutions

1. Nextcloud Photos with Recognize: The Photos app, paired with Recognize, is intended to serve as a simple tool for photo viewing and basic face detection. Unfortunately, its performance as a photo viewer is underwhelming, with sluggish response times and limited functionality. Moreover, Recognize struggles with face detection accuracy, failing to identify many faces within photo libraries. This makes it an unreliable option for users looking to organize their photos by people.
2. Nextcloud Memories with Facedetection: The Memories app is a better photo viewer compared to Photos, providing a more polished user experience directly within the Nextcloud interface. However, its face detection capabilities are limited by its reliance on either Recognize or Facedetection. While Facedetection boasts a robust face detection stack, its classification abilities are poor. This leads to a frustrating user experience where new photo uploads often result in previously clustered faces being misidentified, requiring manual reassignment and reorganization of clusters.

Why Immich.app is the Superior Alternative

Immich.app emerges as a powerful alternative to the existing photo management solutions in Nextcloud. It offers several key advantages:

1. Advanced Facial Recognition and Classification: Immich.app outperforms both Recognize and Facedetection in terms of facial recognition and classification with the use of insightface, RetinaFace&ArcFace with the antelepov2 model. The app not only accurately detects faces in photos but also reliably classifies them, ensuring that new photos are correctly categorized with minimal user intervention. This reduces the need for constant manual corrections, making photo management much more efficient and enjoyable.
2. Superior Media Viewing Experience: As a media viewer, Immich.app offers a richer feature set than both Photos and Memories. It supports various media formats, provides a smooth browsing experience, and includes features like automatic photo enhancements, slideshows, and more. These functionalities make it a versatile and powerful tool for managing and viewing media collections.
3. Hardware Acceleration for Enhanced Performance: Immich.app supports hardware acceleration, which can significantly improve performance, especially when handling large media libraries or high-resolution content. This is a critical feature that sets it apart from the default Nextcloud photo management options, which can struggle with performance issues as the size of the media library grows.

Integrating Immich.app with Nextcloud

To leverage the full potential of Immich.app, it can be integrated with Nextcloud, allowing it to access all media files stored within the Nextcloud environment. By configuring Immich.app as a media server, users can utilize its superior features and capabilities directly with their existing Nextcloud-stored media. This integration process involves setting up the Immich.app server and pointing it to the media directories managed by Nextcloud. Once set up, Immich.app can serve as the primary media viewer, offering seamless access to all photos and videos along with advanced face detection and classification.

Integrating Immich.app into a Nextcloud setup transforms the way media is managed and viewed. It offers a significant upgrade over the existing Photos and Memories apps by providing reliable and accurate face detection, better media viewing features, and enhanced performance through hardware acceleration. This makes it an ideal solution for anyone looking to streamline their photo management workflow and ensure their media is organized efficiently and accurately. With Immich.app, Nextcloud users can enjoy a more powerful, user-friendly, and reliable media management experience.

How to install the Immich.app on an already installed Nextcloud server as external files.

Advantages of Mounting Under /var/www/nextcloud/data/files-external:

Organized Structure:

The files-external folder is an established location for external file storage in Nextcloud. This ensures a clear separation between internal Nextcloud data and external files or data sources.

Access Management:

Nextcloud already has mechanisms in place for managing and accessing external storage locations. Adding external file storage via this path can facilitate management and integration.

Security and Access Control:

By mounting in a specific subfolder like files-external, you can ensure that permissions and access are better controlled, as this folder is specifically intended for external sources.

Avoidance of Conflicts:

Mounting the Immich folder in a separate subfolder within Nextcloud can help avoid conflicts or confusion between internal Nextcloud data and data managed by Immich.

Implementation:

To mount the Immich folder under /var/www/nextcloud/data/files-external, you can adjust the docker-compose.yml file accordingly.

Here are the steps:

1. In the Nextcloud, download and install the App “External Storage”.
2. Proceed to the Immich.app Download and install instructions here:
   https://immich.app/docs/install/docker-compose
3. When you downloaded the docker-compose.yml file, edit it accordingly to the code box below, but with your own folder and user structure.
4. Continue with the Immich.app installation
5. Add an “external library” within the Immich app and add your “/var/www/nextcloud/data” folder
6. Scan the external library within the Immich.app and enjoy a great media server with excellent features.
7. Please note that I mounted the Immich.App with read-only support for testing purposes. Mount it regular or with rw if you want to manage (delete and upload) your media through the Immich.App here. Also: you have to make a file scan in Nextcloud then to make the Immich uploaded media available in the Nextcloud environment here.
   To mount Immich in rw mode use the line:
   

Example docker-compose.yml file:

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    user: "www-data" #same system user for nextcloud and immich
    volumes:
      - /var/www/nextcloud/data/files-external:/usr/src/app/upload  # Mount auf files-external, adjust this to your own path.
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    user: "www-data"
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:e3b17ba9479deec4b7d1eeec1548a253acc5374d68d3b27937fcfe4df8d18c7e
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c", "shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always

volumes:
  model-cache:

In this file, the user www-data runs the Immich processes, same as the Nextcloud server. So you do not have to set rights or access and the Immich app simply can use the same source folder as the Nextcloud for the Images when it is added as external library within the Immich app. Also the storage folder (in my case a 4 TB harddrive is mouted in the Immich app as storage, but under the external files management of the Nextcloud installation, which you could also make availble within the Nextcloud, so you could see files that originate from Immich. I left this part out though, since I use the Nextcloud mobile app to upload all pictures from my Phone, and the Nextcloud desktop app to upload everything else, including my Sony and Fuji camera files.

Edit the .env file of Immich to mount your Nextcloud media folder like this: (please change your mounts according to your folder structure!)

# The location where your uploaded files are stored
UPLOAD_LOCATION=/var/www/nextcloud/data/files_external
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
# TZ=Etc/UTC

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release
#IMMICH_VERSION=v1.113.1

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

NVIDIA_VISIBLE_DEVICES=all

GPU? Intel, Nvidia? Use the hardware for blazing fast video transcoding and machine learning.

In the Immich app, hardware transcoding for videos and GPU-accelerated machine learning can be enabled to improve performance, especially when dealing with large libraries of images and videos.

Steps to Enable Hardware Transcoding and GPU for Machine Learning:

1. Ensure CUDA and GPU support:
   • The server hosting Immich needs a compatible GPU with CUDA support. Ensure the GPU drivers and CUDA (version 12.6 or higher) are properly installed.
   • Verify the GPU installation using nvidia-smi to confirm the CUDA version and GPU availability.
2. Configure Docker for GPU:
   • Immich runs on Docker, and to leverage GPU capabilities, the Docker container must be configured with NVIDIA runtime support. Install the nvidia-docker package and use the --gpus flag when running Immich’s Docker containers.
   • Example Docker command:bashCode kopierendocker run --gpus all -d -p 3003:3003 immich-server
   This command ensures that the Immich app uses the available GPU resources.
3. Enable video transcoding:
   • Immich supports hardware-accelerated transcoding for videos via ffmpeg, which can use your GPU for encoding and decoding tasks.
   • Make sure that the ffmpeg build on your system is compiled with NVIDIA hardware support (e.g., --enable-nvenc for NVIDIA encoding).
   • Inside Immich, the transcoding settings can be configured in the environment variables, setting it to use the GPU for handling video files.
4. Machine learning for face and object detection:
   • Immich leverages GPU-accelerated machine learning for face and object detection tasks using models that are compatible with CUDA.
   • Once the Docker container is running with GPU support, Immich will automatically use the GPU to perform machine learning tasks, speeding up the face detection and recognition processes significantly compared to CPU-only configurations.

By enabling GPU support for both transcoding and machine learning, you can significantly reduce processing times and improve the overall efficiency of handling large media libraries within the Immich app.

If you want to only add the media files of one Nextcloud user, then edit the storage to the specific user path like “/var/www/nextcloud/data/user/files/Photos”. If you do this for selected users, you can control the access within the Immich app, where the owner of an external library can access only his own files and not the external library of another user. In this case, you have to add every needed Nextcloud user storage folder to the docker-compose.yml files and add different external libraries within the Immich app.

Now that works a bit better

Mounting the Immich folder under /var/www/nextcloud/data/files-external provides a clear and structured way to integrate Immich data into Nextcloud without affecting Nextcloud’s internal structure. This approach simplifies management and ensures better organization of your data. Immich now can be used as media viewer with all its nice features, including GPS data, face recognition, face classification and clustering and much more.

There are probably many more integration parts that could be done here, let me know if you had other or better ideas.