9 Commits

Author SHA1 Message Date
3ae91523d8 Update README.md 2025-01-04 04:06:18 +03:00
535c790c39 Add pipeline status badge 2025-01-04 04:04:46 +03:00
73aa5c4ac2 Update README.md 2025-01-04 04:01:36 +03:00
077f920986 Create go.yml for workflows 2025-01-04 03:56:18 +03:00
77443e2369 Update README.md 2025-01-04 03:53:32 +03:00
9acb554dd4 Cleanup, add ability to skip initial full sync 2025-01-04 03:31:33 +03:00
e3f8f7b8c8 Update README.md, fix cow 2025-01-04 03:30:55 +03:00
62ca9f2378 Update README.md 2025-01-04 02:31:44 +03:00
f73d612666 Restore COW on copy for linux 2025-01-04 02:28:26 +03:00
7 changed files with 124 additions and 56 deletions

28
.github/workflows/go.yml vendored Normal file
View File

@@ -0,0 +1,28 @@
# This workflow will build a golang project
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-go
name: Go
on:
push:
branches: [ "master" ]
pull_request:
branches: [ "master" ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: '1.22'
- name: Build
run: go build -v ./...
- name: Test
run: go test -v ./...

View File

@@ -1,25 +1,23 @@
# Simple photo cataloguer
[![Go](https://github.com/derfenix/photocatalog/actions/workflows/go.yml/badge.svg)](https://github.com/derfenix/photocatalog/actions/workflows/go.yml)
# Effortless Photo Organizer
Just copy/hardlink photos (or video, or any other files) from one place to
another, separating them in sub-directories like `$ROOT/year/month/day/`.
### TL;DR
I have a smartphone, I have a Syncthing ~~uugh... SmartThing~~ and all photos
from smartphone nicely synced to my PC without my attention. But I can't just
keep all photos in synced folder: if I'll clean my phone memory - all photos
from pc will be cleaned too. I need to not forget copy files in another
place before cleaning phone's memory. Also, I can't just drop all photos in
one dir - I will not find anything there later, and a folder with thousands
photos looks like a bad idea from either side.
So I create this tool in one evening. All it does - copy (or create hardlinks for)
files from one place to another, creating basic date-aware directories
structure for that files.
I use a smartphone along with Syncthing to seamlessly sync all my photos to my PC without any manual effort. However, there's a catch: I can't keep all my photos in the synced folder indefinitely. If I clear my phone's memory, the photos on my PC get deleted as well. To avoid this, I need to remember to copy the files to another location before cleaning up my phone.
Simply dumping all my photos into one folder isn't a solution either — finding anything later would be a nightmare, and a folder with thousands of unsorted photos is far from ideal.
To address these issues, I created this tool in just one evening. Its primary purpose is to copy (or create hardlinks for) files from one location to another, while organizing them into a simple, date-based directory structure.
This tool was built for personal use and has been serving me well for quite some time without any problems. However, if you encounter any issues, feel free to report them — Id be happy to help.
## Installing
```bash
go install github.com/derfenix/photocatalog@latest
go install github.com/derfenix/photocatalog/v2@latest
```
Optionally you could copy created binary from the GO's bin path to
system or user $PATH, e.g. /usr/local/bin/.
@@ -27,44 +25,63 @@ system or user $PATH, e.g. /usr/local/bin/.
sudo cp ${GOPATH}/bin/photocatalog /usr/local/bin/photocatalog
```
## Supported formats
At this moment supported jpeg files with filled exif data or any other
files but with names matching pattern `yyymmdd_HHMMSS.ext`. Such
names format applied by android's camera software (I guess all cams
use this format, fix me if I'm wrong).
## Migrating from v0.*
There is no support for changing names format without modifying source code
at this time.
TODO
## Organization modes
Next organization modes supported:
- **copy** — copy files to target root. Make COW (using syscall) if FS supports it.
- **hardlink** — create hardlink to the source file instead of copying.
The best choice if source and target are in same partition for compatibility
and resource usage, but we can't chmod target files, because of original file mode will
be changed too.
- **move** — moves original files to new place.
- **symlink** — create a symlink at the target for the source files.
## Supported formats
At this moment supported jpeg and tiff files with filled exif data and any other
files but with names matching pattern `yyymmdd_HHMMSS.ext` with optional suffixes after a timestamp.
Such names format applied by the Android's camera software (I guess all cams
use this format, fix me if I'm wrong).
Jpeg/Tiff files without modification date if exif will be fallen back to the name parsing.
No able to change names format without modifying source code for now. Just because
I have reasons to believe that this format is the most popular for the application use cases.
But let me know if you need different timestamp formats support.
## Usage
### One-shot
#### Copy files (make a COW if fs supports it)
#### Copy files
```bash
photocalog -mode copy -target ./photos/ ./sync/photos/*
photocalog -mode copy -target ./photos/ -source ./sync/photos/
```
#### Create hardlinks (only withing one disk partition)
#### Create hardlinks
```bash
photocalog -mode hardlink -target ./photos/ ./sync/photos/*
photocalog -mode hardlink -target ./photos/ -source ./sync/photos/
```
or
```bash
photocalog -target ./photos/ ./sync/photos/*
photocalog -target ./photos/ -source ./sync/photos/*
```
### Monitor
#### Copy files (make a COW if fs supports it)
### Watch mode
#### Copy files
```bash
photocalog -mode copy -target ./photos -monitor ./sync/photos/*
photocalog -mode copy -target ./photos -watch -source ./sync/photos/
```
#### Create hardlinks (only withing one disk partition)
#### Create hardlinks
```bash
photocalog -mode hardlink -target ./photos/ -monitor ./sync/photos/
photocalog -mode hardlink -target ./photos/ -watch -source ./sync/photos/
```
or
```bash
photocalog -target ./photos/ -monitor ./sync/photos/
photocalog -target ./photos/ -watch -source ./sync/photos/
```
## Install and run monitor service
@@ -89,6 +106,6 @@ under corresponding sub-dir.
### Why this tool was created if there is awesome XXX tool?
I had two good reasons:
1. I wanted
1. I wish
2. I can

View File

@@ -50,8 +50,10 @@ func (a *Application) Start(ctx context.Context, wg *sync.WaitGroup) error {
org = org.WithOverwrite()
}
if err := org.FullSync(ctx); err != nil {
return fmt.Errorf("full sync: %w", err)
if !a.config.SkipFullSync {
if err := org.FullSync(ctx); err != nil {
return fmt.Errorf("full sync: %w", err)
}
}
if a.config.Watch {

View File

@@ -15,13 +15,14 @@ const (
)
type Config struct {
SourceDir string
TargetDir string
Mode Mode
Overwrite bool
DirMode uint64
FileMode uint64
Watch bool
SourceDir string
TargetDir string
Mode Mode
Overwrite bool
DirMode uint64
FileMode uint64
Watch bool
SkipFullSync bool
}
func (c *Config) Validate() error {
@@ -33,9 +34,13 @@ func (c *Config) Validate() error {
return fmt.Errorf("target dir is required")
}
if !slices.Contains([]Mode{ModeHardlink, ModeSymlink, ModeMove, ModeSymlink}, c.Mode) {
if !slices.Contains([]Mode{ModeHardlink, ModeSymlink, ModeMove, ModeCopy}, c.Mode) {
return fmt.Errorf("invalid mode %s", c.Mode)
}
if c.SkipFullSync && !c.Watch {
return fmt.Errorf("skip full sync and watch disabled — nothing to do")
}
return nil
}

View File

@@ -5,12 +5,18 @@ import (
"io"
"log"
"os"
"runtime"
"sync/atomic"
"golang.org/x/sys/unix"
)
var cowDisabled = atomic.Bool{}
type Copy struct{}
func (c Copy) PlaceIt(sourcePath, targetPath string, mode os.FileMode) error {
targetFile, err := os.OpenFile(targetPath, os.O_TRUNC|os.O_RDWR|os.O_CREATE, mode)
targetFile, err := os.OpenFile(targetPath, os.O_TRUNC|os.O_WRONLY|os.O_CREATE, mode)
if err != nil {
return fmt.Errorf("open target file: %w", err)
}
@@ -28,6 +34,17 @@ func (c Copy) PlaceIt(sourcePath, targetPath string, mode os.FileMode) error {
_ = sourceFile.Close()
}()
// Try to do a COW.
if runtime.GOOS == "linux" && !cowDisabled.Load() {
if err := unix.IoctlFileClone(int(targetFile.Fd()), int(sourceFile.Fd())); err == nil {
return nil
} else {
log.Println(fmt.Errorf("COW attempt for %s failed: %w", targetPath, err))
log.Println("Disabling COW until restart")
cowDisabled.Store(true)
}
}
copySize, err := io.Copy(targetFile, sourceFile)
if err != nil {
return fmt.Errorf("copy source file: %w", err)

View File

@@ -17,6 +17,11 @@ import (
"github.com/derfenix/photocatalog/v2/internal/metadata"
)
const (
defaultDirMode = 0o777
defaultFileMode = 0o644
)
type MetaExtractor interface {
Extract(_ string, data io.Reader) (metadata.Metadata, error)
}
@@ -31,8 +36,8 @@ func NewOrganizer(mode Mode, source, target string) *Organizer {
sourceDir: source,
targetDir: target,
dirMode: 0777,
fileMode: 0644,
dirMode: defaultDirMode,
fileMode: defaultFileMode,
metaExtractors: map[string]MetaExtractor{
"": &metadata.Default{},
@@ -268,6 +273,7 @@ func (o *Organizer) BuildTargetPath(sourcePath string, meta metadata.Metadata) (
o.targetDir,
strconv.Itoa(meta.Created.Year()),
strconv.Itoa(int(meta.Created.Month())),
strconv.Itoa(meta.Created.Day()),
sourcePath,
)

17
main.go
View File

@@ -33,25 +33,18 @@ func main() {
}
func loadCfg() application.Config {
cfg := application.Config{
SourceDir: "",
TargetDir: "",
Mode: "",
Overwrite: false,
DirMode: 0,
FileMode: 0,
Watch: false,
}
cfg := application.Config{}
flag.StringVar(&cfg.SourceDir, "source", "", "Source directory")
flag.StringVar(&cfg.TargetDir, "target", "", "Target directory")
flag.BoolVar(&cfg.Overwrite, "overwrite", false, "Overwrite existing files")
flag.BoolVar(&cfg.Watch, "watch", false, "Watch for changes in the source directory")
flag.BoolVar(&cfg.Watch, "watch", true, "Watch for changes in the source directory")
flag.BoolVar(&cfg.SkipFullSync, "skip-full-sync", false, "Skip full sync at startup")
var dirMode string
var fileMode string
flag.StringVar(&dirMode, "dirmode", "0777", "Mode bits for directories can be created while syncing")
flag.StringVar(&fileMode, "filemode", "0644", "Mode bits for files created while syncing (not applicable for hardlink mode)")
flag.StringVar(&dirMode, "dir-mode", "0777", "Mode bits for directories can be created while syncing")
flag.StringVar(&fileMode, "file-mode", "0644", "Mode bits for files created while syncing (not applicable for hardlink mode)")
var mode string
flag.StringVar(&mode, "mode", "hardlink", "Mode")