15 Commits

Author SHA1 Message Date
bcaeba5ea6 Update README.md 2025-01-06 19:54:31 +03:00
9248a9a84d Update README.md, refactoring 2025-01-06 18:15:22 +03:00
f0a8abb380 Update README.md 2025-01-06 15:18:06 +03:00
fb1ab2f8b5 Update README.md 2025-01-04 22:50:22 +03:00
bdd3eee69f Update README.md 2025-01-04 19:39:59 +03:00
8e315ff557 Update README.md 2025-01-04 19:37:23 +03:00
3ae91523d8 Update README.md 2025-01-04 04:06:18 +03:00
535c790c39 Add pipeline status badge 2025-01-04 04:04:46 +03:00
73aa5c4ac2 Update README.md 2025-01-04 04:01:36 +03:00
077f920986 Create go.yml for workflows 2025-01-04 03:56:18 +03:00
77443e2369 Update README.md 2025-01-04 03:53:32 +03:00
9acb554dd4 Cleanup, add ability to skip initial full sync 2025-01-04 03:31:33 +03:00
e3f8f7b8c8 Update README.md, fix cow 2025-01-04 03:30:55 +03:00
62ca9f2378 Update README.md 2025-01-04 02:31:44 +03:00
f73d612666 Restore COW on copy for linux 2025-01-04 02:28:26 +03:00
7 changed files with 188 additions and 94 deletions

28
.github/workflows/go.yml vendored Normal file
View File

@@ -0,0 +1,28 @@
# This workflow will build a golang project
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-go
name: Go
on:
push:
branches: [ "master" ]
pull_request:
branches: [ "master" ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: '1.22'
- name: Build
run: go build -v ./...
- name: Test
run: go test -v ./...

161
README.md
View File

@@ -1,94 +1,129 @@
# Simple photo cataloguer
# Effortless Photo Organizer
Just copy/hardlink photos (or video, or any other files) from one place to
another, separating them in sub-directories like `$ROOT/year/month/day/`.
[![Go](https://github.com/derfenix/photocatalog/actions/workflows/go.yml/badge.svg)](https://github.com/derfenix/photocatalog/actions/workflows/go.yml)
### TL;DR
A simple tool to organize your photos, videos, or other files by copying or hardlinking them into a date-based directory structure like `$ROOT/year/month/day/`.
I have a smartphone, I have a Syncthing ~~uugh... SmartThing~~ and all photos
from smartphone nicely synced to my PC without my attention. But I can't just
keep all photos in synced folder: if I'll clean my phone memory - all photos
from pc will be cleaned too. I need to not forget copy files in another
place before cleaning phone's memory. Also, I can't just drop all photos in
one dir - I will not find anything there later, and a folder with thousands
photos looks like a bad idea from either side.
So I create this tool in one evening. All it does - copy (or create hardlinks for)
files from one place to another, creating basic date-aware directories
structure for that files.
## TL;DR
I use a smartphone and Syncthing to automatically sync my photos to my PC. However, if I clean up my phone's memory, the synced photos on my PC are deleted as well.
Dumping everything into one folder wasn't an option — finding anything later would be a nightmare.
To avoid this, I needed a solution to back up and organize my photos without manual effort. So, I built this tool in one evening to solve the problem. It has worked flawlessly for me and might help you too. If you encounter any issues, feel free to open a ticket — I'll do my best to assist.
## Installation
Install the tool via `go`:
## Installing
```bash
go install github.com/derfenix/photocatalog@latest
go install github.com/derfenix/photocatalog/v2@latest
```
Optionally you could copy created binary from the GO's bin path to
system or user $PATH, e.g. /usr/local/bin/.
Optionally, copy the binary to a directory in your system or user's `$PATH` (e.g., `/usr/local/bin`):
```bash
sudo cp ${GOPATH}/bin/photocatalog /usr/local/bin/photocatalog
```
## Supported formats
At this moment supported jpeg files with filled exif data or any other
files but with names matching pattern `yyymmdd_HHMMSS.ext`. Such
names format applied by android's camera software (I guess all cams
use this format, fix me if I'm wrong).
## Organization Modes
There is no support for changing names format without modifying source code
at this time.
The tool supports the following organization modes:
- **copy** — Copies files to the target directory. If the filesystem supports it, uses Copy-on-Write (COW) for efficiency (via FICLONE ioctl call).
- **hardlink** — Creates hardlinks to the source files, saving disk space. Ideal if the source and target are on the same partition, though file permissions remain linked to the original.
- **move** — Moves files from the source to the target directory.
- **symlink** — Creates symbolic links at the target pointing to the source files.
## Supported Formats
- **JPEG and TIFF files** with valid EXIF metadata.
- Files named in the format `yyyymmdd_HHMMSS.ext` (optionally with suffixes after the timestamp) (e.g., `20230101_123456.jpg`). This format is common in Android cameras and other devices.
If a file lacks EXIF data, the tool falls back to parsing the filename.
Currently, the timestamp format is not customizable. Let me know if support for additional formats is required.
## Usage
### One-shot
#### Copy files (make a COW if fs supports it)
```bash
photocalog -mode copy -target ./photos/ ./sync/photos/*
Arguments
```
-dir-mode string
Mode bits for directories can be created while syncing (default "0777")
-file-mode string
Mode bits for files created while syncing (not applicable for hardlink mode) (default "0644")
-mode string
Organazing mode (default "hardlink")
-overwrite
Overwrite existing files
-skip-full-sync
Skip full sync at startup
-source string
Source directory
-target string
Target directory
-watch
Watch for changes in the source directory (default true)
```
#### Create hardlinks (only withing one disk partition)
```bash
photocalog -mode hardlink -target ./photos/ ./sync/photos/*
```
or
```bash
photocalog -target ./photos/ ./sync/photos/*
`-skip-full-sync` and `-watch` are not compatible.
`-source` and `-target` are required.
## Examples
### One-Time Run
#### Copy Files
```shell
photocatalog -mode copy -target ./photos/ -source ./sync/photos/
```
### Monitor
#### Copy files (make a COW if fs supports it)
```bash
photocalog -mode copy -target ./photos -monitor ./sync/photos/*
#### Create Hardlinks
```shell
photocatalog -mode hardlink -target ./photos/ -source ./sync/photos/
```
#### Create hardlinks (only withing one disk partition)
```bash
photocalog -mode hardlink -target ./photos/ -monitor ./sync/photos/
```
or
```bash
photocalog -target ./photos/ -monitor ./sync/photos/
### Watch Mode
Enable continuous monitoring of a source directory:
#### Copy Files
```shell
photocatalog -mode copy -target ./photos -watch -source ./sync/photos/
```
## Install and run monitor service
#### Create Hardlinks
```shell
photocatalog -mode hardlink -target ./photos/ -watch -source ./sync/photos/
```
### Systemd
```bash
## Running as a Service
### Systemd Setup
Install and configure the service:
```shell
sh ./init/install_service.sh systemd
```
This command will install unit file, create stub for its config and open
editor to allow you edit configuration. Config file stored at
`$HOME/.config/photocatalog`.
Then enable and start service
```bash
This will:
1. Install a systemd unit file.
2. Create a configuration stub at `$HOME/.config/photocatalog`.
3. Open the config file for editing.
Enable and start the service:
```shell
systemctl --user enable --now photocatalog
```
That's all. Now, if any file will be placed in directory, specified as `MONITOR`
in config file, this file will be copied or hardlinked into the target dir
under corresponding sub-dir.
Now, files added to the monitored directory (`MONITOR` in the config) will automatically be organized into the target directory under the corresponding subdirectories.
## FAQ
### Why this tool was created if there is awesome XXX tool?
I had two good reasons:
1. I wanted
2. I can
### Why did you create this tool when awesome tool XXX already exists?
Two reasons:
1. I wanted to.
2. I could.

View File

@@ -50,9 +50,11 @@ func (a *Application) Start(ctx context.Context, wg *sync.WaitGroup) error {
org = org.WithOverwrite()
}
if !a.config.SkipFullSync {
if err := org.FullSync(ctx); err != nil {
return fmt.Errorf("full sync: %w", err)
}
}
if a.config.Watch {
if err := org.Watch(ctx, wg); err != nil {

View File

@@ -22,6 +22,7 @@ type Config struct {
DirMode uint64
FileMode uint64
Watch bool
SkipFullSync bool
}
func (c *Config) Validate() error {
@@ -33,9 +34,13 @@ func (c *Config) Validate() error {
return fmt.Errorf("target dir is required")
}
if !slices.Contains([]Mode{ModeHardlink, ModeSymlink, ModeMove, ModeSymlink}, c.Mode) {
if !slices.Contains([]Mode{ModeHardlink, ModeSymlink, ModeMove, ModeCopy}, c.Mode) {
return fmt.Errorf("invalid mode %s", c.Mode)
}
if c.SkipFullSync && !c.Watch {
return fmt.Errorf("skip full sync and watch disabled — nothing to do")
}
return nil
}

View File

@@ -5,12 +5,18 @@ import (
"io"
"log"
"os"
"runtime"
"sync/atomic"
"golang.org/x/sys/unix"
)
var cowDisabled = atomic.Bool{}
type Copy struct{}
func (c Copy) PlaceIt(sourcePath, targetPath string, mode os.FileMode) error {
targetFile, err := os.OpenFile(targetPath, os.O_TRUNC|os.O_RDWR|os.O_CREATE, mode)
targetFile, err := os.OpenFile(targetPath, os.O_TRUNC|os.O_WRONLY|os.O_CREATE, mode)
if err != nil {
return fmt.Errorf("open target file: %w", err)
}
@@ -28,6 +34,17 @@ func (c Copy) PlaceIt(sourcePath, targetPath string, mode os.FileMode) error {
_ = sourceFile.Close()
}()
// Try to do a COW.
if runtime.GOOS == "linux" && !cowDisabled.Load() {
if err := unix.IoctlFileClone(int(targetFile.Fd()), int(sourceFile.Fd())); err == nil {
return nil
} else {
log.Println(fmt.Errorf("COW attempt for %s failed: %w", targetPath, err))
log.Println("Disabling COW until restart")
cowDisabled.Store(true)
}
}
copySize, err := io.Copy(targetFile, sourceFile)
if err != nil {
return fmt.Errorf("copy source file: %w", err)

View File

@@ -17,6 +17,11 @@ import (
"github.com/derfenix/photocatalog/v2/internal/metadata"
)
const (
defaultDirMode = 0o777
defaultFileMode = 0o644
)
type MetaExtractor interface {
Extract(_ string, data io.Reader) (metadata.Metadata, error)
}
@@ -31,8 +36,8 @@ func NewOrganizer(mode Mode, source, target string) *Organizer {
sourceDir: source,
targetDir: target,
dirMode: 0777,
fileMode: 0644,
dirMode: defaultDirMode,
fileMode: defaultFileMode,
metaExtractors: map[string]MetaExtractor{
"": &metadata.Default{},
@@ -175,7 +180,9 @@ func (o *Organizer) FullSync(ctx context.Context) error {
}
if err := o.processFile(path); err != nil {
return err
log.Printf("Process file `%s` failed: %s", path, err.Error())
return nil
}
return nil
@@ -198,7 +205,7 @@ func (o *Organizer) getMetaForPath(fp string) (metadata.Metadata, error) {
meta, err := o.getMetadata(fp, file)
if err != nil {
return metadata.Metadata{}, fmt.Errorf("get metadata: %w", err)
return metadata.Metadata{}, fmt.Errorf("get metadatas: %w", err)
}
return meta, nil
@@ -268,6 +275,7 @@ func (o *Organizer) BuildTargetPath(sourcePath string, meta metadata.Metadata) (
o.targetDir,
strconv.Itoa(meta.Created.Year()),
strconv.Itoa(int(meta.Created.Month())),
strconv.Itoa(meta.Created.Day()),
sourcePath,
)

31
main.go
View File

@@ -33,28 +33,23 @@ func main() {
}
func loadCfg() application.Config {
cfg := application.Config{
SourceDir: "",
TargetDir: "",
Mode: "",
Overwrite: false,
DirMode: 0,
FileMode: 0,
Watch: false,
}
cfg := application.Config{}
flag.StringVar(&cfg.SourceDir, "source", "", "Source directory")
flag.StringVar(&cfg.TargetDir, "target", "", "Target directory")
flag.BoolVar(&cfg.Overwrite, "overwrite", false, "Overwrite existing files")
flag.BoolVar(&cfg.Watch, "watch", false, "Watch for changes in the source directory")
flag.BoolVar(&cfg.Watch, "watch", true, "Watch for changes in the source directory")
flag.BoolVar(&cfg.SkipFullSync, "skip-full-sync", false, "Skip full sync at startup")
var dirMode string
var fileMode string
flag.StringVar(&dirMode, "dirmode", "0777", "Mode bits for directories can be created while syncing")
flag.StringVar(&fileMode, "filemode", "0644", "Mode bits for files created while syncing (not applicable for hardlink mode)")
var (
dirMode string
fileMode string
mode string
)
var mode string
flag.StringVar(&mode, "mode", "hardlink", "Mode")
flag.StringVar(&dirMode, "dir-mode", "0777", "Mode bits for directories can be created while syncing")
flag.StringVar(&fileMode, "file-mode", "0644", "Mode bits for files created while syncing (not applicable for hardlink mode)")
flag.StringVar(&mode, "mode", "hardlink", "Organizing mode")
flag.Parse()
@@ -64,11 +59,15 @@ func loadCfg() application.Config {
cfg.DirMode, err = strconv.ParseUint(dirMode, 8, 32)
if err != nil {
log.Println("Parse -dir-mode failed:", err)
cfg.DirMode = 0o777
}
cfg.FileMode, err = strconv.ParseUint(fileMode, 8, 32)
if err != nil {
log.Println("Parse -file-mode failed:", err)
cfg.DirMode = 0o644
}