From cc55f55dab114392f8d0cddc7283fe91cfa79f14 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Stephan=20D=C3=BCsterhaupt?= Date: Sun, 5 Jul 2026 18:39:27 +0200 Subject: [PATCH] Initial project commit --- .gitignore | 21 ++ CHANGELOG.md | 20 ++ LICENSE | 21 ++ README.md | 172 ++++++++++ tr2gf.sh | 916 +++++++++++++++++++++++++++++++++++++++++++++++++++ vars.example | 81 +++++ 6 files changed, 1231 insertions(+) create mode 100644 .gitignore create mode 100644 CHANGELOG.md create mode 100644 LICENSE create mode 100644 README.md create mode 100755 tr2gf.sh create mode 100644 vars.example diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..7e7d00c --- /dev/null +++ b/.gitignore @@ -0,0 +1,21 @@ +# Ignore everything by default +* + +# Allow directories +!*/ + +# Allow Git internals +!/.gitignore + +# Allow shell scripts +!*.sh + +# Allow Markdown files +!*.md + +# Allow vars.example file +!vars.example + +# Allow LICENSE and CHANGELOG +!LICENSE +!CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..943ede2 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,20 @@ +# Changelog + +All notable changes to this project will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## 1.0.0 - 2026-07-05 + +### Added + +- Added `tr2gf.sh` as a POSIX-compliant Trade Republic transaction export converter for Ghostfolio. +- Added support for `ALL` and filtered selector modes. +- Added config-driven symbol, source and code mappings loaded from a local `vars` file. +- Added CLI symbol overrides via `-S`, `--symbol` and `--symbol=VALUE`. +- Added count-only mode for transaction inspection without export file generation. +- Added runtime verification for required files, directories, environment variables and tool dependencies. +- Added structured logging helpers and batch-aware notices. +- Added AWK-based CSV parsing, transaction filtering, cancellation matching and Ghostfolio CSV export generation. +- Added default source handling for Yahoo-backed instruments and CoinGecko-backed crypto symbols. diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..568e34e --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Stephan Düsterhaupt + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md new file mode 100644 index 0000000..c5af0ef --- /dev/null +++ b/README.md @@ -0,0 +1,172 @@ +# tr2gf + +Trade Republic transaction export converter for Ghostfolio. + +`tr2gf.sh` converts a Trade Republic transaction export CSV into one Ghostfolio-compatible CSV file per selected ISIN or symbol. The script is POSIX-compliant, supports config-driven symbol and source mappings, optional CLI overrides, count-only runs and per-symbol export file naming based on the latest matching transaction date. + +## Outline + +- [Features](#features) +- [Repository layout](#repository-layout) +- [Requirements](#requirements) +- [Installation](#installation) +- [Configuration](#configuration) +- [Usage](#usage) +- [Examples](#examples) +- [Output format](#output-format) +- [Logging and troubleshooting](#logging-and-troubleshooting) +- [License](#license) +- [Authors](#authors) +- [Project Home](#project-home) + +## Features + +- Convert a Trade Republic CSV export into one Ghostfolio-compatible CSV file per matching symbol. +- Export `BUY` and `SELL` transactions and suppress matching `BUY_CANCELLED` records. +- Load runtime defaults and symbol/source mappings from a local `vars` file. +- Override symbol codes per selector on the command line via `-S` or `--symbol`. +- Run in count-only mode without writing export files. +- Keep the implementation portable with `#!/bin/sh` and standard Unix tools such as `awk`, `sed`, `grep` and `date`. + +## Repository layout + +```text +. +├── .gitignore +├── tr2gf.sh +├── vars.example +├── README.md +├── CHANGELOG.md +└── LICENSE +``` + +## Requirements + +The script checks for the presence of `awk`, `sed`, `grep` and `date` at runtime and it expects a readable source CSV plus an output directory when not running in count-only mode. + +## Installation + +1. Clone the repository. +2. Copy `vars.example` to `vars`. +3. Adjust the runtime defaults and symbol mappings for the local environment. +4. Make `tr2gf.sh` executable. +5. Run a help or count-only command first to validate the setup. + +```sh +git clone https://github.com/sduesterhaupt/tr2gf.git + +cd tr2gf + +cp vars.example vars +chmod 0750 tr2gf.sh +chmod 0640 vars + +./tr2gf.sh help +``` + +The `tr2gf.sh` script is executable only for the repository owner, and the `vars` file is readable by owner and group to keep configuration and symbol mappings under tighter control. + +## Configuration + +The script initializes built-in defaults for the source file, output directory, account name, logging and default data sources, then loads overrides from a local `vars` file if present. + +Supported configurable keys loaded from `vars` are: + +- `TR2GF_SOURCE_FILE` +- `TR2GF_OUTPUT_DIR` +- `TR2GF_ACCOUNT_NAME` +- `TR2GF_YAHOO_SOURCE` +- `TR2GF_CRYPTO_SOURCE` +- `TR2GF_LOG_DIR` +- `TR2GF_LOG_FILE` +- `TR2GF_LOG_LEVEL` +- `TR2GF_BATCH` + +Symbol mappings are read as colon-separated entries in the form `SYMBOL:CODE[:SOURCE]`. When no source is set, known crypto symbols default to the configured crypto data source and all other symbols default to the configured Yahoo source. + +## Usage + +General command form: + +```sh +./tr2gf.sh [options] TARGET [target-options] +``` + +Global options implemented by the script: + +- `-f FILE` define a specific Trade Republic source CSV. +- `-o DIR` define a specific export output directory. +- `-a NAME` define the account name written to the export CSV. +- `-c` count matching `BUY` and `SELL` transactions only. +- `-h`, `--help` show usage information. + +Target modes implemented by the script: + +- `ALL` export all matching `BUY` and `SELL` transactions found in the source CSV. +- `ISIN [target-opts]` export transactions only for the specified selectors (e.g. ISINs and crypto tickers). + +Target options implemented by the script (per selector): + +- `-S SYMBOL` +- `--symbol SYMBOL` +- `--symbol=SYMBOL` + +## Examples + +Export all supported transactions into one file per symbol: + +```sh +./tr2gf.sh ALL +``` + +Count transactions for selected selectors without writing files: + +```sh +./tr2gf.sh -c IE00B4L5Y983 BTC +``` + +Export a selected symbol with a CLI override for the target code: + +```sh +./tr2gf.sh IE00BK5BQT80 -S VWCE.DE +``` + +Write exports into a dedicated directory and set the Ghostfolio account label: + +```sh +./tr2gf.sh -o ./exports -a "Trade Republic" ALL +``` + +## Output format + +Each generated file starts with the Ghostfolio-compatible header `Date,Code,DataSource,Currency,Price,Quantity,Action,Fee,Account,Note` and the filename is prefixed with the latest matching transaction date in `YYMMDD` form. + +The generated `Note` field uses the pattern `ISIN `, while the `Action` column is mapped to `buy` or `sell` and fees are written as absolute numeric values. + +## Logging and troubleshooting + +The script contains a file-based logging system with severity levels from `0` to `5`, configurable via `TR2GF_LOG_LEVEL` and writes to `TR2GF_LOG_FILE` when enabled. + +For troubleshooting, useful first checks are: + +```sh +./tr2gf.sh help +./tr2gf.sh -c ALL +sed -n '1,5p' ./TradeRepublic_TransactionExport.csv +``` + +Adjust the filename in the example above if your Trade Republic export has a date prefix or a different name. + +Typical failure points are a missing source CSV, a wrong output directory, absent required commands or mismatched CSV headers such as missing `date`, `type`, `symbol`, `name`, `shares`, `price`, `fee` or `currency` columns. + +## License + +This project is published under the MIT License. See `LICENSE` for details. + +## Authors + +- [Stephan Düsterhaupt](xmpp:me@jabber.stephanduesterhaupt.de) + +## Project Home + +Project Home: [https://dev.town-square.de/sduesterhaupt/tr2gf](https://dev.town-square.de/sduesterhaupt/tr2gf) diff --git a/tr2gf.sh b/tr2gf.sh new file mode 100755 index 0000000..04de267 --- /dev/null +++ b/tr2gf.sh @@ -0,0 +1,916 @@ +#!/bin/sh + +############################################################################### + +# tr2gf.sh + +# Trade Republic transaction export converter + +# This script converts a Trade Republic transaction export CSV into one +# Ghostfolio-compatible CSV file per selected ISIN or symbol. It supports +# config-driven symbol/code mappings, optional CLI overrides, transaction +# filtering, and export file generation with the latest transaction date prefix. + +# Features: +# - Export all matching BUY and SELL transactions per symbol +# - Load configuration defaults and symbol/code mappings from a vars file +# - Override per-ISIN symbol mappings via CLI parameters +# - Count matching transactions without writing export files +# - POSIX-compatible operation for repeatable local or batch execution + +# Authors: Stephan Düsterhaupt +# Copyright (c) 2026 Stephan Düsterhaupt +# License: MIT + +# Project Home: https://dev.town-square.de/sduesterhaupt/tr2gf + +############################################################################### + +# MIT License + +# Copyright (c) 2026 Stephan Düsterhaupt + +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: + +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. + +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +set -eu + + +# SDuesterhaupt: 2026-07-05 - Write DEBUG entry logs for function tracing. +_log_enter() { + _log "Enter the function '$1()'..." 1 + return 0 +} #=> _log_enter() + + +# SDuesterhaupt: 2026-07-05 - Write DEBUG exit logs for function tracing. +_log_leave() { + _log "Leave the function '$1()'..." 1 + return 0 +} #=> _log_leave() + + +# SDuesterhaupt: 2026-07-05 - Show general usage information and an overview +# of available target modes. +_usage() { + _log_enter "_usage" + + err_source="Not defined: vars autodetect failed or disabled" + current_vars="$TR2GF/vars" + [ -f "$current_vars" ] || current_vars="$err_source" + source_file=${TR2GF_SOURCE_FILE:-$err_source} + output_dir=${TR2GF_OUTPUT_DIR:-$err_source} + account_name=${TR2GF_ACCOUNT_NAME:-$err_source} + + printf '%s\n' \ +"tr2gf usage and overview" \ +"" \ +"USAGE: tr2gf.sh [options] TARGET [target-options]" \ +"" \ +"A list of supported target modes is shown below. To get detailed usage and help, run:" \ +" tr2gf.sh help TARGET" \ +"" \ +"For a listing of options that may be supplied before the target, use:" \ +" tr2gf.sh help options" \ +"" \ +"Target modes:" \ +" ALL" \ +" ISIN [ target-opts ]" \ +"" \ +"RUNTIME STATUS (exports would use these locations and defaults)" \ +" VARS: $current_vars" \ +" SOURCE: $source_file" \ +" OUTPUT: $output_dir" \ +" ACCOUNT: $account_name" \ +"" + + _log_leave "_usage" + return 0 +} #=> _usage() + + +# SDuesterhaupt: 2026-07-05 - Show target-specific help text or global option help. +# When called without arguments, show general usage or target-specific help. +_cmd_help() { + _log_enter "_cmd_help" + text="" + opts="" + + case "$1" in + ALL) + text="ALL\n Export all matching BUY and SELL transactions found in the source CSV." + ;; + ISIN) + text="ISIN [ target-opts ]\n Export transactions only for the specified ISIN selectors." + opts="-S SYMBOL\n --symbol=VALUE\n symbol=VALUE" + ;; + options) + _opt_usage + _log_leave "_cmd_help" + return 0 + ;; + "") + _usage + _log_leave "_cmd_help" + return 0 + ;; + *) + text="Unknown help target: '$1' (run without arguments for an overview)" + ;; + esac + + [ -n "$text" ] && printf '%b\n' "$text" + + if [ -n "$opts" ]; then + printf '\n%s\n' "target-opts is an optional set of target options from this list:" + printf '%b\n' "$opts" + fi + + _log_leave "_cmd_help" + return 0 +} #=> _cmd_help() + + +# SDuesterhaupt: 2026-07-05 - Print descriptions of global command-line options. +_opt_usage() { + _log_enter "_opt_usage" + printf '%s\n' \ +"tr2gf Global Option Flags" \ +"" \ +"The following options may be provided before the target mode. Options specified" \ +"at runtime override built-in defaults and values loaded from the local vars" \ +"file. Unless noted, non-empty values to options are mandatory." \ +"" \ +"General options:" \ +" -f FILE define a specific Trade Republic source CSV" \ +" -o DIR define a specific export output directory" \ +" -a NAME define the account name written to the export CSV" \ +" -c count matching BUY/SELL transactions per symbol only" \ +" -h, --help show general usage information" \ +"" \ +"Target options:" \ +" -S SYMBOL override Code for the immediately preceding ISIN" \ +" --symbol=VALUE override Code for the immediately preceding ISIN" \ +"" + + _log_leave "_opt_usage" + return 0 +} #=> _opt_usage() + + +# SDuesterhaupt: 2026-07-05 - Abort execution with an error message and exit code. +_die() { + msg=$1 + code=${2:-1} + + printf '\n%s\n%s\n' "tr2gf error:" "$msg" >&2 + + if [ "$code" -ne 0 ] && [ -n "${TR2GF_LOG_FILE:-}" ]; then + ts=$(date '+%Y-%m-%d %H:%M:%S') + printf '%s tr2gf [ERROR] %s\n' "$ts" "$msg" >>"$TR2GF_LOG_FILE" + fi + + exit "$code" +} #=> _die() + + +# SDuesterhaupt: 2026-07-05 - Print an informational notice unless batch mode is enabled. +_notice() { + if [ -z "${TR2GF_BATCH:-}" ]; then + printf '\n%s\n' "$1" + fi + + return 0 +} #=> _notice() + + +# SDuesterhaupt: 2026-07-05 - Verify essential runtime variables are defined. +# Checks the presence of critical env vars for tr2gf runtime. +_vars_source_check() { + _log_enter "_vars_source_check" + + [ -n "${TR2GF:-}" ] || _die "TR2GF env-var undefined" 5 + [ -n "${TR2GF_SOURCE_FILE:-}" ] || _die "TR2GF_SOURCE_FILE env-var undefined" 5 + [ -n "${TR2GF_OUTPUT_DIR:-}" ] || _die "TR2GF_OUTPUT_DIR env-var undefined" 5 + [ -n "${TR2GF_ACCOUNT_NAME:-}" ] || _die "TR2GF_ACCOUNT_NAME env-var undefined" 5 + [ -n "${TR2GF_YAHOO_SOURCE:-}" ] || _die "TR2GF_YAHOO_SOURCE env-var undefined" 5 + [ -n "${TR2GF_CRYPTO_SOURCE:-}" ] || _die "TR2GF_CRYPTO_SOURCE env-var undefined" 5 + [ -n "${TR2GF_COUNT_ONLY:-}" ] || _die "TR2GF_COUNT_ONLY env-var undefined" 5 + [ -n "${TR2GF_SYMBOL_MAPS+x}" ] || _die "TR2GF_SYMBOL_MAPS env-var undefined" 5 + + _log_leave "_vars_source_check" + return 0 +} #=> _vars_source_check() + + +# SDuesterhaupt: 2026-07-05 - Verify tr2gf runtime directories, files and tools. +# Checks existence of required paths and command availability. +_verify_runtime_init() { + _log_enter "_verify_runtime_init" + help_note="Run tr2gf.sh without targets for usage and target help." + + _vars_source_check + + [ -f "$TR2GF_SOURCE_FILE" ] || _die "TR2GF_SOURCE_FILE missing. $help_note" 5 + if [ "$TR2GF_COUNT_ONLY" -eq 0 ]; then + [ -d "$TR2GF_OUTPUT_DIR" ] || _die "TR2GF_OUTPUT_DIR missing. $help_note" 5 + fi + + command -v awk >/dev/null 2>&1 || _die "awk not available. $help_note" 5 + command -v sed >/dev/null 2>&1 || _die "sed not available. $help_note" 5 + command -v grep >/dev/null 2>&1 || _die "grep not available. $help_note" 5 + command -v date >/dev/null 2>&1 || _die "date not available. $help_note" 5 + + _log "Runtime checks passed for source '$TR2GF_SOURCE_FILE'." 2 + + _log_leave "_verify_runtime_init" + return 0 +} #=> _verify_runtime_init() + + +# SDuesterhaupt: 2026-07-05 - Ensure the log directory exists before logging starts. +_init_log() { + _log_enter "_init_log" + if [ ! -d "$TR2GF_LOG_DIR" ]; then + mkdir -p "$TR2GF_LOG_DIR" 2>/dev/null \ + || _die "Could not create log directory '$TR2GF_LOG_DIR'" 1 + fi + + _log_leave "_init_log" + return 0 +} #=> _init_log() + + +# SDuesterhaupt: 2026-07-05 - Load configuration defaults and local vars file. +_vars_setup() { + _log_enter "_vars_setup" + prog_file=$0 + prog_dir=$(dirname "$prog_file") + + _set_var TR2GF "$prog_dir" + + _set_var TR2GF_SOURCE_FILE "$TR2GF/TradeRepublic_TransactionExport.csv" + _set_var TR2GF_OUTPUT_DIR "$TR2GF/exports" + _set_var TR2GF_ACCOUNT_NAME "DEFAULT" + _set_var TR2GF_COUNT_ONLY 0 + _set_var TR2GF_YAHOO_SOURCE "YAHOO" + _set_var TR2GF_CRYPTO_SOURCE "COINGECKO" + _set_var TR2GF_SYMBOL_MAPS "" + + # Logging + _set_var TR2GF_LOG_DIR "$TR2GF/log" + _set_var TR2GF_LOG_FILE "$TR2GF_LOG_DIR/tr2gf.log" + _set_var TR2GF_LOG_LEVEL 2 + _set_var TR2GF_BATCH "" + + if [ -f "$TR2GF/vars" ]; then + _load_vars_file "$TR2GF/vars" + _notice "Note: using tr2gf configuration from: $TR2GF/vars" + _log "Loaded configuration from: $TR2GF/vars" 2 + else + _log "No local vars file found. Using built-in defaults." 2 + fi + + _log "Runtime defaults prepared." 2 + + _log_leave "_vars_setup" + return 0 +} #=> _vars_setup() + + +# SDuesterhaupt: 2026-07-05 - Set a variable to a default value if it is unset. +_set_var() { + var=$1 + shift + eval "$var=\${$var:-\"\$*\"}" + eval "export $var" + return 0 +} #=> _set_var() + + +# SDuesterhaupt: 2026-07-05 - Write a message to the tr2gf log file. +# +# 0 -> OFF (silent mode) +# 1 -> DEBUG +# 2 -> INFO +# 3 -> WARN +# 4 -> ERROR +# 5 -> CRITICAL +# +# @param1: Message string to log. +# @param2: Severity level (0-5) where higher is more severe. +# Writes message with timestamp to log file. +# Prints to stderr if level >= 3. +_log() { + msg=$1 + level=${2:-0} + + set -- "" "DEBUG" "INFO" "WARN" "ERROR" "CRITICAL" + shift + level_name=$([ "$level" -ge 1 ] && eval "printf '%s' \${$level}") + level_tag="[$level_name]" + level_tag_len=${#level_tag} + pad_width=10 + pad_count=$(( pad_width - level_tag_len )) + [ "$pad_count" -lt 1 ] && pad_count=1 + pad=$(printf '%*s' "$pad_count" '') + + if [ -n "${TR2GF_LOG_LEVEL:-}" ] && [ "$level" -ge "$TR2GF_LOG_LEVEL" ] 2>/dev/null && [ -n "$msg" ] && [ -n "${TR2GF_LOG_FILE:-}" ]; then + ts=$(date '+%Y-%m-%d %H:%M:%S') + printf '%s tr2gf %s%s%s\n' "$ts" "$level_tag" "$pad" "$msg" >>"$TR2GF_LOG_FILE" + fi + + if [ "$level" -ge 3 ] 2>/dev/null && [ -n "$msg" ] && [ -z "${TR2GF_BATCH:-}" ]; then + printf '%s\n' "$msg" >&2 + fi + + return 0 +} #=> _log() + + +# SDuesterhaupt: 2026-07-05 - Strip one pair of outer quotes from a config value. +_strip_outer_quotes() { + value=$1 + + case "$value" in + \"*\") + value=${value#\"} + value=${value%\"} + ;; + \'*\') + value=${value#\'} + value=${value%\'} + ;; + esac + + printf '%s\n' "$value" + return 0 +} #=> _strip_outer_quotes() + + +# SDuesterhaupt: 2026-07-05 - Detect whether a symbol should use crypto defaults. +_is_crypto_symbol() { + case "$1" in + BTC|ETH|SOL|ADA|XRP|DOT|DOGE|AVAX|LINK|MATIC) + return 0 + ;; + *) + return 1 + ;; + esac +} #=> _is_crypto_symbol() + + +# SDuesterhaupt: 2026-07-05 - Load supported config values and mapping entries from a vars file. +_load_vars_file() { + _log_enter "_load_vars_file" + vars_file=$1 + [ -f "$vars_file" ] || _die "vars file not found: $vars_file" 1 + + while IFS= read -r raw_line || [ -n "$raw_line" ]; do + line=$raw_line + case "$line" in + ''|'#'*) + continue + ;; + esac + + line=$(printf '%s' "$line" | sed 's/^[[:space:]]*//; s/[[:space:]]*$//') + [ -n "$line" ] || continue + + case "$line" in + *=*) + key=${line%%=*} + value=${line#*=} + key=$(printf '%s' "$key" | sed 's/^[[:space:]]*//; s/[[:space:]]*$//') + value=$(printf '%s' "$value" | sed 's/^[[:space:]]*//; s/[[:space:]]*$//') + value=$(_strip_outer_quotes "$value") + case "$key" in + TR2GF_SOURCE_FILE|TR2GF_OUTPUT_DIR|TR2GF_ACCOUNT_NAME|TR2GF_YAHOO_SOURCE|TR2GF_CRYPTO_SOURCE|TR2GF_LOG_DIR|TR2GF_LOG_FILE|TR2GF_LOG_LEVEL|TR2GF_BATCH) + eval "$key=\$value" + eval "export $key" + ;; + *) + ;; + esac + ;; + *:*) + symbol=$(printf '%s' "$line" | awk -F: '{print $1}') + code=$(printf '%s' "$line" | awk -F: '{print $2}') + source=$(printf '%s' "$line" | awk -F: '{if (NF >= 3) {sub(/^[^:]*:[^:]*:/, ""); print} else {print ""}}') + symbol=$(printf '%s' "$symbol" | sed 's/^[[:space:]]*//; s/[[:space:]]*$//') + code=$(printf '%s' "$code" | sed 's/^[[:space:]]*//; s/[[:space:]]*$//') + source=$(printf '%s' "$source" | sed 's/^[[:space:]]*//; s/[[:space:]]*$//') + [ -n "$symbol" ] || continue + [ -n "$code" ] || continue + if [ -z "$source" ]; then + if _is_crypto_symbol "$symbol"; then + source=$TR2GF_CRYPTO_SOURCE + else + source=$TR2GF_YAHOO_SOURCE + fi + fi + if [ -n "$TR2GF_SYMBOL_MAPS" ]; then + TR2GF_SYMBOL_MAPS="${TR2GF_SYMBOL_MAPS}|${symbol}:${source}:${code}" + else + TR2GF_SYMBOL_MAPS="${symbol}:${source}:${code}" + fi + export TR2GF_SYMBOL_MAPS + _log "Loaded symbol override: ${symbol} -> ${code} (${source})" 1 + ;; + *) + ;; + esac + done < "$vars_file" + + _log_leave "_load_vars_file" + return 0 +} #=> _load_vars_file() + + +# SDuesterhaupt: 2026-07-05 - Print active runtime configuration values. +_print_runtime_info() { + _log_enter "_print_runtime_info" + current_vars="$TR2GF/vars" + [ -f "$current_vars" ] || current_vars="not loaded" + + printf '%s\n' \ + "Runtime configuration" \ + "" \ + "VARS FILE: $current_vars" \ + "SOURCE FILE: $TR2GF_SOURCE_FILE" \ + "OUTPUT DIR: $TR2GF_OUTPUT_DIR" \ + "ACCOUNT NAME: $TR2GF_ACCOUNT_NAME" \ + "DEFAULT YAHOO SOURCE: $TR2GF_YAHOO_SOURCE" \ + "DEFAULT CRYPTO SOURCE: $TR2GF_CRYPTO_SOURCE" \ + "COUNT ONLY: $TR2GF_COUNT_ONLY" \ + "" + + _log_leave "_print_runtime_info" + return 0 +} #=> _print_runtime_info() + + +# SDuesterhaupt: 2026-07-05 - Append a symbol selector to the target list. +_append_symbol() { + _log_enter "_append_symbol" + isin=$1 + + if [ -n "$SYMBOLS" ]; then + SYMBOLS="${SYMBOLS}|${isin}" + else + SYMBOLS=$isin + fi + + _log "Queued symbol selector: $isin" 1 + + _log_leave "_append_symbol" + return 0 +} #=> _append_symbol() + + +# SDuesterhaupt: 2026-07-05 - Append a source/code override entry. +_append_override() { + _log_enter "_append_override" + isin=$1 + source=$2 + code=$3 + entry="${isin}:${source}:${code}" + + if [ -n "$CLI_OVERRIDES" ]; then + CLI_OVERRIDES="${CLI_OVERRIDES}|${entry}" + else + CLI_OVERRIDES=$entry + fi + + _log "Queued CLI override: ${isin} -> ${code} (${source})" 2 + + _log_leave "_append_override" + return 0 +} #=> _append_override() + + +_vars_setup +_init_log + +while [ $# -gt 0 ]; do + opt=${1%%=*} + + case "$opt" in + -f) + shift + [ $# -gt 0 ] || _die "-f requires a file path" 2 + TR2GF_SOURCE_FILE=$1 + shift + ;; + -o) + shift + [ $# -gt 0 ] || _die "-o requires a directory path" 2 + TR2GF_OUTPUT_DIR=$1 + shift + ;; + -a) + shift + [ $# -gt 0 ] || _die "-a requires a value" 2 + TR2GF_ACCOUNT_NAME=$1 + shift + ;; + -c) + TR2GF_COUNT_ONLY=1 + shift + ;; + -h|--help) + _usage + exit 0 + ;; + --) + shift + break + ;; + -S|--symbol|--symbol=*) + break + ;; + -*) + _die "unknown option: $1" 2 + ;; + *) + break + ;; + esac +done + +case "${1:-}" in + help) + shift + _cmd_help "${1:-}" + exit 0 + ;; + -h|--help) + _usage + exit 0 + ;; +esac + +if [ $# -lt 1 ]; then + _print_runtime_info >&2 + _usage >&2 + exit 2 +fi + +if [ "$TR2GF_COUNT_ONLY" -eq 0 ]; then + mkdir -p -- "$TR2GF_OUTPUT_DIR" + _log "Ensured output directory exists: $TR2GF_OUTPUT_DIR" 2 +else + _log "Count-only mode enabled. No export files will be written." 2 +fi + +_verify_runtime_init + +MODE=FILTER +SYMBOLS= +CLI_OVERRIDES= + +if [ "$1" = "ALL" ]; then + if [ $# -ne 1 ]; then + _die "ALL cannot be combined with selectors or symbol overrides" 2 + fi + MODE=ALL + shift + set -- +else + pending_isin= + pending_override_set=0 + pending_override= + + while [ $# -gt 0 ]; do + case "$1" in + --symbol=*) + [ -n "$pending_isin" ] || _die "$1 must follow an ISIN selector" 2 + [ "$pending_override_set" -eq 0 ] || _die "multiple symbol overrides for ISIN $pending_isin" 2 + pending_override_set=1 + pending_override=${1#*=} + shift + ;; + -S|--symbol) + [ -n "$pending_isin" ] || _die "$1 must follow an ISIN selector" 2 + [ "$pending_override_set" -eq 0 ] || _die "multiple symbol overrides for ISIN $pending_isin" 2 + shift + [ $# -gt 0 ] || _die "-S/--symbol requires a value" 2 + pending_override_set=1 + pending_override=$1 + shift + ;; + *) + if [ -n "$pending_isin" ]; then + _append_symbol "$pending_isin" + if [ "$pending_override_set" -eq 1 ]; then + if _is_crypto_symbol "$pending_isin"; then + _append_override "$pending_isin" "$TR2GF_CRYPTO_SOURCE" "$pending_override" + else + _append_override "$pending_isin" "$TR2GF_YAHOO_SOURCE" "$pending_override" + fi + fi + fi + pending_isin=$1 + pending_override_set=0 + pending_override= + shift + ;; + esac + done + + if [ -n "$pending_isin" ]; then + _append_symbol "$pending_isin" + if [ "$pending_override_set" -eq 1 ]; then + if _is_crypto_symbol "$pending_isin"; then + _append_override "$pending_isin" "$TR2GF_CRYPTO_SOURCE" "$pending_override" + else + _append_override "$pending_isin" "$TR2GF_YAHOO_SOURCE" "$pending_override" + fi + fi + fi +fi + +MERGED_MAPS=$TR2GF_SYMBOL_MAPS +if [ -n "$CLI_OVERRIDES" ]; then + _log "Applying CLI symbol overrides: $CLI_OVERRIDES" 2 + if [ -n "$MERGED_MAPS" ]; then + MERGED_MAPS="${MERGED_MAPS}|${CLI_OVERRIDES}" + else + MERGED_MAPS=$CLI_OVERRIDES + fi +fi + +_log "Execution mode: $MODE" 2 +if [ -n "$SYMBOLS" ]; then + _log "Selected symbols: $SYMBOLS" 2 +fi + +_log "Starting AWK conversion for source '$TR2GF_SOURCE_FILE'." 2 +_log "Merged symbol/source mappings: $MERGED_MAPS" 1 + +TMP_AWK="${TMPDIR:-/tmp}/tr_export_$$.awk" +trap 'rm -f "$TMP_AWK"' EXIT HUP INT TERM + +cat >"$TMP_AWK" <<'AWK' +function csv_parse(line, out, i, n, ch, nextch, field, quoted) { + delete out + n = 0 + field = "" + quoted = 0 + for (i = 1; i <= length(line); i++) { + ch = substr(line, i, 1) + if (quoted) { + if (ch == "\"") { + nextch = substr(line, i + 1, 1) + if (nextch == "\"") { + field = field "\"" + i++ + } else { + quoted = 0 + } + } else { + field = field ch + } + } else { + if (ch == "\"") { + quoted = 1 + } else if (ch == ",") { + out[++n] = field + field = "" + } else { + field = field ch + } + } + } + out[++n] = field + return n +} + +function csv_quote(s, t) { + t = s + gsub(/"/, """", t) + return "\"" t "\"" +} + +function trim(s) { + sub(/^[[:space:]]+/, "", s) + sub(/[[:space:]]+$/, "", s) + return s +} + +function sanitize_filename(s, t) { + t = s + gsub(/[[:space:]]+/, "-", t) + gsub(/[^[:alnum:].-]+/, "-", t) + gsub(/-+/, "-", t) + sub(/^-/, "", t) + sub(/-$/, "", t) + if (t == "") t = "unnamed" + return t +} + +function absval(x) { + return (x < 0) ? -x : x +} + +function shares_abs(s, x) { + x = trim(s) + 0 + if (x < 0) x = -x + return sprintf("%.10f", x) +} + +function crypto_default_code(symbol) { + if (symbol == "BTC") return "bitcoin" + else if (symbol == "ETH") return "ethereum" + else if (symbol == "SOL") return "solana" + else if (symbol == "ADA") return "cardano" + else if (symbol == "XRP") return "ripple" + else if (symbol == "DOT") return "polkadot" + else if (symbol == "DOGE") return "dogecoin" + else if (symbol == "AVAX") return "avalanche-2" + else if (symbol == "LINK") return "chainlink" + else if (symbol == "MATIC") return "matic-network" + return symbol +} + +function detect_data_source(symbol) { + if (symbol in forced_source) return forced_source[symbol] + if (symbol == "BTC" || symbol == "ETH" || symbol == "SOL" || symbol == "ADA" || symbol == "XRP" || symbol == "DOT" || symbol == "DOGE" || symbol == "AVAX" || symbol == "LINK" || symbol == "MATIC") return default_crypto_source + return default_yahoo_source +} + +function map_code(symbol, source, key) { + key = symbol SUBSEP source + if (key in forced_code && forced_code[key] != "") return forced_code[key] + if (source == default_crypto_source) return crypto_default_code(symbol) + return symbol +} + +function append_record(symbol, line) { + if (!(symbol in record_count)) { + record_count[symbol] = 0 + records[symbol] = "" + } + record_count[symbol]++ + if (record_count[symbol] > 1) records[symbol] = records[symbol] "\n" + records[symbol] = records[symbol] line +} + +BEGIN { + want_all = (mode == "ALL") ? 1 : 0 + + if (!want_all && symbols != "") { + nsel = split(symbols, selected, "|") + for (i = 1; i <= nsel; i++) wanted[selected[i]] = 1 + } + + if (maps != "") { + nmap = split(maps, map_entries, "|") + for (i = 1; i <= nmap; i++) { + n = split(map_entries[i], parts, ":") + if (n >= 3) { + symbol_key = parts[1] + source_key = parts[2] + code_value = substr(map_entries[i], length(parts[1]) + length(parts[2]) + 3) + forced_source[symbol_key] = source_key + forced_code[symbol_key SUBSEP source_key] = code_value + } + } + } +} + +NR == 1 { + nf = csv_parse($0, hdr) + for (i = 1; i <= nf; i++) col[trim(hdr[i])] = i + + required["date"] = 1 + required["type"] = 1 + required["symbol"] = 1 + required["name"] = 1 + required["shares"] = 1 + required["price"] = 1 + required["fee"] = 1 + required["currency"] = 1 + + for (k in required) { + if (!(k in col)) { + printf("Error: missing required column: %s\n", k) > "/dev/stderr" + exit 1 + } + } + next +} + +{ + nf = csv_parse($0, row) + date = trim(row[col["date"]]) + typ = trim(row[col["type"]]) + symbol = trim(row[col["symbol"]]) + name = trim(row[col["name"]]) + shares = trim(row[col["shares"]]) + price = trim(row[col["price"]]) + fee = trim(row[col["fee"]]) + currency = trim(row[col["currency"]]) + typ_u = toupper(typ) + + if (symbol == "") next + if (!want_all && !(symbol in wanted)) next + + cancel_key = symbol SUBSEP date SUBSEP shares_abs(shares) + if (typ_u == "BUY_CANCELLED") { + cancelled[cancel_key]++ + next + } + if (typ_u != "BUY" && typ_u != "SELL") next + if (typ_u == "BUY" && cancelled[cancel_key] > 0) { + cancelled[cancel_key]-- + next + } + + count[symbol]++ + last_name[symbol] = name + safe_name[symbol] = sanitize_filename(name) + + if (count_only == 1) next + + action = (typ_u == "BUY") ? "buy" : "sell" + fee_value = (fee == "") ? 0 : absval(fee + 0) + note = "ISIN " symbol + data_source = detect_data_source(symbol) + export_code = map_code(symbol, data_source) + + line = sprintf("%s,%s,%s,%s,%s,%s,%s,%.2f,%s,%s", date, export_code, data_source, currency, price, shares, action, fee_value, csv_quote(account_name), csv_quote(note)) + append_record(symbol, line) + + if (!(symbol in latest_date) || date > latest_date[symbol]) latest_date[symbol] = date +} + +END { + found = 0 + for (s in count) found = 1 + + if (count_only == 1) { + if (want_all) { + for (s in count) printf("%s;%d;%s\n", s, count[s], last_name[s]) + } else { + for (i = 1; i <= nsel; i++) { + s = selected[i] + printf("%s;%d;%s\n", s, ((s in count) ? count[s] : 0), ((s in last_name) ? last_name[s] : "")) + } + } + exit 0 + } + + if (!found) { + print "No matching BUY/SELL transactions found." > "/dev/stderr" + exit 1 + } + + for (s in records) { + prefix = latest_date[s] + gsub(/-/, "", prefix) + prefix = substr(prefix, 3, 6) + filename = output_dir "/" prefix "_" s "_" safe_name[s] ".csv" + print "Date,Code,DataSource,Currency,Price,Quantity,Action,Fee,Account,Note" > filename + print records[s] >> filename + close(filename) + printf("written;%s;%d;%s\n", filename, count[s], last_name[s]) + } +} +AWK + +if awk \ + -v mode="$MODE" \ + -v symbols="$SYMBOLS" \ + -v maps="$MERGED_MAPS" \ + -v count_only="$TR2GF_COUNT_ONLY" \ + -v output_dir="$TR2GF_OUTPUT_DIR" \ + -v account_name="$TR2GF_ACCOUNT_NAME" \ + -v default_yahoo_source="$TR2GF_YAHOO_SOURCE" \ + -v default_crypto_source="$TR2GF_CRYPTO_SOURCE" \ + -f "$TMP_AWK" \ + "$TR2GF_SOURCE_FILE" +then + _log "AWK conversion finished successfully." 2 + exit 0 +else + rc=$? + _log "AWK conversion failed with exit code $rc." 4 + exit "$rc" +fi diff --git a/vars.example b/vars.example new file mode 100644 index 0000000..8f46b99 --- /dev/null +++ b/vars.example @@ -0,0 +1,81 @@ +# tr2gf configuration example +# Copy this file to 'vars' and adjust as needed. +# Warning: do not edit vars.example directly! + +# ------------------------------------------------------------------ +# GENERAL +# ------------------------------------------------------------------ + +# Trade Republic source CSV used as default input file +#TR2GF_SOURCE_FILE=./TradeRepublic_TransactionExport.csv + +# Output directory for generated Ghostfolio import files +#TR2GF_OUTPUT_DIR=./exports + +# Account name written to the export CSV +#TR2GF_ACCOUNT_NAME=DEFAULT + +# Batch mode: +# Leave empty for interactive/normal console output. +# Set to any non-empty string to suppress notices on stdout. +#TR2GF_BATCH= + +# ------------------------------------------------------------------ +# DATA SOURCES +# ------------------------------------------------------------------ + +# Default data source for non-crypto assets +#TR2GF_YAHOO_SOURCE=YAHOO + +# Default data source for supported crypto assets +#TR2GF_CRYPTO_SOURCE=COINGECKO + +# ------------------------------------------------------------------ +# LOGGING +# ------------------------------------------------------------------ + +# Directory for tr2gf log files +#TR2GF_LOG_DIR=./log + +# Main log file +#TR2GF_LOG_FILE=./log/tr2gf.log + +# Log level controls verbosity of logging output: +# 0 = off : Disable all logging output +# 1 = debug : Detailed diagnostics for troubleshooting and development +# 2 = info : Informational messages about normal operations +# 3 = warning : Warnings about unusual but non-fatal conditions +# 4 = error : Errors affecting functionality +# 5 = critical : Severe failures requiring immediate attention +#TR2GF_LOG_LEVEL=2 + +# ------------------------------------------------------------------ +# SYMBOL / CODE OVERRIDES +# ------------------------------------------------------------------ + +# Format: +# SYMBOL:CODE:SOURCE +# +# SOURCE is optional. If missing or empty, tr2gf selects +# TR2GF_CRYPTO_SOURCE for supported crypto symbols and +# TR2GF_YAHOO_SOURCE for all other symbols. +# +# Examples: +# IE00BJ5JPG56:ICHN.AS:YAHOO +# IE00BHZRQZ17:FLXI.L:YAHOO +# BTC:bitcoin:COINGECKO +# ETH:ethereum:COINGECKO +# IE00BJ5JPG56:ICHN.AS +# BTC:bitcoin: +# +# Notes: +# - Empty lines and lines starting with '#' are ignored. +# - CLI parameters override values from this file. +# - For YAHOO assets without an override, the ISIN/symbol is used as Code. +# - For supported crypto assets without an override, built-in CoinGecko mappings apply. + +#IE00BJ5JPG56:ICHN.AS:YAHOO +#IE00BHZRQZ17:FLXI.L:YAHOO +#BTC:bitcoin:COINGECKO +#ETH:ethereum:COINGECKO +#SOL:solana: