11 Commits

Author SHA1 Message Date
bdd3eee69f Update README.md 2025-01-04 19:39:59 +03:00
8e315ff557 Update README.md 2025-01-04 19:37:23 +03:00
3ae91523d8 Update README.md 2025-01-04 04:06:18 +03:00
535c790c39 Add pipeline status badge 2025-01-04 04:04:46 +03:00
73aa5c4ac2 Update README.md 2025-01-04 04:01:36 +03:00
077f920986 Create go.yml for workflows 2025-01-04 03:56:18 +03:00
77443e2369 Update README.md 2025-01-04 03:53:32 +03:00
9acb554dd4 Cleanup, add ability to skip initial full sync 2025-01-04 03:31:33 +03:00
e3f8f7b8c8 Update README.md, fix cow 2025-01-04 03:30:55 +03:00
62ca9f2378 Update README.md 2025-01-04 02:31:44 +03:00
f73d612666 Restore COW on copy for linux 2025-01-04 02:28:26 +03:00
7 changed files with 145 additions and 88 deletions

28
.github/workflows/go.yml vendored Normal file
View File

@@ -0,0 +1,28 @@
# This workflow will build a golang project
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-go
name: Go
on:
push:
branches: [ "master" ]
pull_request:
branches: [ "master" ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: '1.22'
- name: Build
run: go build -v ./...
- name: Test
run: go test -v ./...

132
README.md
View File

@@ -1,94 +1,100 @@
# Simple photo cataloguer # Effortless Photo Organizer
Just copy/hardlink photos (or video, or any other files) from one place to [![Go](https://github.com/derfenix/photocatalog/actions/workflows/go.yml/badge.svg)](https://github.com/derfenix/photocatalog/actions/workflows/go.yml)
another, separating them in sub-directories like `$ROOT/year/month/day/`.
### TL;DR A simple tool to organize your photos, videos, or other files by copying or hardlinking them into a date-based directory structure like `$ROOT/year/month/day/`.
I have a smartphone, I have a Syncthing ~~uugh... SmartThing~~ and all photos ## TL;DR
from smartphone nicely synced to my PC without my attention. But I can't just
keep all photos in synced folder: if I'll clean my phone memory - all photos
from pc will be cleaned too. I need to not forget copy files in another
place before cleaning phone's memory. Also, I can't just drop all photos in
one dir - I will not find anything there later, and a folder with thousands
photos looks like a bad idea from either side.
So I create this tool in one evening. All it does - copy (or create hardlinks for)
files from one place to another, creating basic date-aware directories
structure for that files.
I use a smartphone and Syncthing to automatically sync my photos to my PC. However, if I clean up my phone's memory, the synced photos on my PC are deleted as well. To avoid this, I needed a solution to back up and organize my photos without manual effort.
Dumping everything into one folder wasn't an option — finding anything later would be a nightmare. So, I built this tool in one evening to solve the problem. It has worked flawlessly for me and might help you too. If you encounter any issues, feel free to open a ticket — I'll do my best to assist.
## Installation
Install the tool via `go`:
## Installing
```bash ```bash
go install github.com/derfenix/photocatalog@latest go install github.com/derfenix/photocatalog/v2@latest
``` ```
Optionally you could copy created binary from the GO's bin path to
system or user $PATH, e.g. /usr/local/bin/. Optionally, copy the binary to a directory in your system or user's `$PATH` (e.g., `/usr/local/bin`):
```bash ```bash
sudo cp ${GOPATH}/bin/photocatalog /usr/local/bin/photocatalog sudo cp ${GOPATH}/bin/photocatalog /usr/local/bin/photocatalog
``` ```
## Supported formats ## Organization Modes
At this moment supported jpeg files with filled exif data or any other
files but with names matching pattern `yyymmdd_HHMMSS.ext`. Such
names format applied by android's camera software (I guess all cams
use this format, fix me if I'm wrong).
There is no support for changing names format without modifying source code The tool supports the following organization modes:
at this time.
- **copy** — Copies files to the target directory. If the filesystem supports it, uses Copy-on-Write (COW) for efficiency.
- **hardlink** — Creates hardlinks to the source files, saving disk space. Ideal if the source and target are on the same partition, though file permissions remain linked to the original.
- **move** — Moves files from the source to the target directory.
- **symlink** — Creates symbolic links at the target pointing to the source files.
## Supported Formats
- **JPEG and TIFF files** with valid EXIF metadata.
- Files named in the format `yyyymmdd_HHMMSS.ext` (optionally with suffixes after the timestamp) (e.g., `20230101_123456.jpg`). This format is common in Android cameras and other devices.
If a file lacks EXIF data, the tool falls back to parsing the filename.
Currently, the timestamp format is not customizable. Let me know if support for additional formats is required.
## Usage ## Usage
### One-shot
#### Copy files (make a COW if fs supports it) ### One-Time Run
```bash
photocalog -mode copy -target ./photos/ ./sync/photos/* #### Copy Files
```shell
photocatalog -mode copy -target ./photos/ -source ./sync/photos/
``` ```
#### Create hardlinks (only withing one disk partition) #### Create Hardlinks
```bash ```shell
photocalog -mode hardlink -target ./photos/ ./sync/photos/* photocatalog -mode hardlink -target ./photos/ -source ./sync/photos/
```
or
```bash
photocalog -target ./photos/ ./sync/photos/*
``` ```
### Monitor ### Watch Mode
#### Copy files (make a COW if fs supports it)
```bash Enable continuous monitoring of a source directory:
photocalog -mode copy -target ./photos -monitor ./sync/photos/*
#### Copy Files
```shell
photocatalog -mode copy -target ./photos -watch -source ./sync/photos/
``` ```
#### Create hardlinks (only withing one disk partition) #### Create Hardlinks
```bash ```shell
photocalog -mode hardlink -target ./photos/ -monitor ./sync/photos/ photocatalog -mode hardlink -target ./photos/ -watch -source ./sync/photos/
```
or
```bash
photocalog -target ./photos/ -monitor ./sync/photos/
``` ```
## Install and run monitor service ## Running as a Service
### Systemd ### Systemd Setup
```bash
Install and configure the service:
```shell
sh ./init/install_service.sh systemd sh ./init/install_service.sh systemd
``` ```
This command will install unit file, create stub for its config and open
editor to allow you edit configuration. Config file stored at
`$HOME/.config/photocatalog`.
Then enable and start service This will:
```bash
1. Install a systemd unit file.
2. Create a configuration stub at `$HOME/.config/photocatalog`.
3. Open the config file for editing.
Enable and start the service:
```shell
systemctl --user enable --now photocatalog systemctl --user enable --now photocatalog
``` ```
That's all. Now, if any file will be placed in directory, specified as `MONITOR`
in config file, this file will be copied or hardlinked into the target dir Now, files added to the monitored directory (`MONITOR` in the config) will automatically be organized into the target directory under the corresponding subdirectories.
under corresponding sub-dir.
## FAQ ## FAQ
### Why this tool was created if there is awesome XXX tool? ### Why did you create this tool when awesome tool XXX already exists?
I had two good reasons: Two reasons:
1. I wanted 1. I wanted to.
2. I can 2. I could.

View File

@@ -50,8 +50,10 @@ func (a *Application) Start(ctx context.Context, wg *sync.WaitGroup) error {
org = org.WithOverwrite() org = org.WithOverwrite()
} }
if err := org.FullSync(ctx); err != nil { if !a.config.SkipFullSync {
return fmt.Errorf("full sync: %w", err) if err := org.FullSync(ctx); err != nil {
return fmt.Errorf("full sync: %w", err)
}
} }
if a.config.Watch { if a.config.Watch {

View File

@@ -15,13 +15,14 @@ const (
) )
type Config struct { type Config struct {
SourceDir string SourceDir string
TargetDir string TargetDir string
Mode Mode Mode Mode
Overwrite bool Overwrite bool
DirMode uint64 DirMode uint64
FileMode uint64 FileMode uint64
Watch bool Watch bool
SkipFullSync bool
} }
func (c *Config) Validate() error { func (c *Config) Validate() error {
@@ -33,9 +34,13 @@ func (c *Config) Validate() error {
return fmt.Errorf("target dir is required") return fmt.Errorf("target dir is required")
} }
if !slices.Contains([]Mode{ModeHardlink, ModeSymlink, ModeMove, ModeSymlink}, c.Mode) { if !slices.Contains([]Mode{ModeHardlink, ModeSymlink, ModeMove, ModeCopy}, c.Mode) {
return fmt.Errorf("invalid mode %s", c.Mode) return fmt.Errorf("invalid mode %s", c.Mode)
} }
if c.SkipFullSync && !c.Watch {
return fmt.Errorf("skip full sync and watch disabled — nothing to do")
}
return nil return nil
} }

View File

@@ -5,12 +5,18 @@ import (
"io" "io"
"log" "log"
"os" "os"
"runtime"
"sync/atomic"
"golang.org/x/sys/unix"
) )
var cowDisabled = atomic.Bool{}
type Copy struct{} type Copy struct{}
func (c Copy) PlaceIt(sourcePath, targetPath string, mode os.FileMode) error { func (c Copy) PlaceIt(sourcePath, targetPath string, mode os.FileMode) error {
targetFile, err := os.OpenFile(targetPath, os.O_TRUNC|os.O_RDWR|os.O_CREATE, mode) targetFile, err := os.OpenFile(targetPath, os.O_TRUNC|os.O_WRONLY|os.O_CREATE, mode)
if err != nil { if err != nil {
return fmt.Errorf("open target file: %w", err) return fmt.Errorf("open target file: %w", err)
} }
@@ -28,6 +34,17 @@ func (c Copy) PlaceIt(sourcePath, targetPath string, mode os.FileMode) error {
_ = sourceFile.Close() _ = sourceFile.Close()
}() }()
// Try to do a COW.
if runtime.GOOS == "linux" && !cowDisabled.Load() {
if err := unix.IoctlFileClone(int(targetFile.Fd()), int(sourceFile.Fd())); err == nil {
return nil
} else {
log.Println(fmt.Errorf("COW attempt for %s failed: %w", targetPath, err))
log.Println("Disabling COW until restart")
cowDisabled.Store(true)
}
}
copySize, err := io.Copy(targetFile, sourceFile) copySize, err := io.Copy(targetFile, sourceFile)
if err != nil { if err != nil {
return fmt.Errorf("copy source file: %w", err) return fmt.Errorf("copy source file: %w", err)

View File

@@ -17,6 +17,11 @@ import (
"github.com/derfenix/photocatalog/v2/internal/metadata" "github.com/derfenix/photocatalog/v2/internal/metadata"
) )
const (
defaultDirMode = 0o777
defaultFileMode = 0o644
)
type MetaExtractor interface { type MetaExtractor interface {
Extract(_ string, data io.Reader) (metadata.Metadata, error) Extract(_ string, data io.Reader) (metadata.Metadata, error)
} }
@@ -31,8 +36,8 @@ func NewOrganizer(mode Mode, source, target string) *Organizer {
sourceDir: source, sourceDir: source,
targetDir: target, targetDir: target,
dirMode: 0777, dirMode: defaultDirMode,
fileMode: 0644, fileMode: defaultFileMode,
metaExtractors: map[string]MetaExtractor{ metaExtractors: map[string]MetaExtractor{
"": &metadata.Default{}, "": &metadata.Default{},
@@ -268,6 +273,7 @@ func (o *Organizer) BuildTargetPath(sourcePath string, meta metadata.Metadata) (
o.targetDir, o.targetDir,
strconv.Itoa(meta.Created.Year()), strconv.Itoa(meta.Created.Year()),
strconv.Itoa(int(meta.Created.Month())), strconv.Itoa(int(meta.Created.Month())),
strconv.Itoa(meta.Created.Day()),
sourcePath, sourcePath,
) )

17
main.go
View File

@@ -33,25 +33,18 @@ func main() {
} }
func loadCfg() application.Config { func loadCfg() application.Config {
cfg := application.Config{ cfg := application.Config{}
SourceDir: "",
TargetDir: "",
Mode: "",
Overwrite: false,
DirMode: 0,
FileMode: 0,
Watch: false,
}
flag.StringVar(&cfg.SourceDir, "source", "", "Source directory") flag.StringVar(&cfg.SourceDir, "source", "", "Source directory")
flag.StringVar(&cfg.TargetDir, "target", "", "Target directory") flag.StringVar(&cfg.TargetDir, "target", "", "Target directory")
flag.BoolVar(&cfg.Overwrite, "overwrite", false, "Overwrite existing files") flag.BoolVar(&cfg.Overwrite, "overwrite", false, "Overwrite existing files")
flag.BoolVar(&cfg.Watch, "watch", false, "Watch for changes in the source directory") flag.BoolVar(&cfg.Watch, "watch", true, "Watch for changes in the source directory")
flag.BoolVar(&cfg.SkipFullSync, "skip-full-sync", false, "Skip full sync at startup")
var dirMode string var dirMode string
var fileMode string var fileMode string
flag.StringVar(&dirMode, "dirmode", "0777", "Mode bits for directories can be created while syncing") flag.StringVar(&dirMode, "dir-mode", "0777", "Mode bits for directories can be created while syncing")
flag.StringVar(&fileMode, "filemode", "0644", "Mode bits for files created while syncing (not applicable for hardlink mode)") flag.StringVar(&fileMode, "file-mode", "0644", "Mode bits for files created while syncing (not applicable for hardlink mode)")
var mode string var mode string
flag.StringVar(&mode, "mode", "hardlink", "Mode") flag.StringVar(&mode, "mode", "hardlink", "Mode")