A capability release. The pipeline could only ever produce a synthetic dataset by opening the real data again, which made repeated generation a repeated privacy event. This adds a reviewed, approved, reusable fitted generator so the real data is opened once.
Gap. Every synthetic dataset required the source data and a human to
run the pipeline. The export bundle's recipe.yaml records the synthesis
configuration, not a fitted generator, so it cannot regenerate data on its
own -- a rerun still meant reopening the real records. Teams that needed a
fresh variation weekly reopened the source weekly.
What landed. A four-object lifecycle, each object distinct and
separately auditable. generator_contract() is the policy: the derived
output schema, roles and settings, the generation_limits() envelope of
permitted seeds, row counts and dataset counts, and a compatibility block,
hashed into an immutable contract_id. freeze_synthesis() fits that
policy against real data in a human-controlled session and writes the
fitted state to a private store, returning a source-free handle that does
not contain the fitted state. approve_generator() is a separate human
sign-off that binds one specific fitted generator revision under the
contract, with a named approver; generation is unavailable until it
succeeds. generate_synthetic() then draws a bounded request against the
approval, returning one dataset or a dataganger_batch of deterministic
variations, each with a sanitized generation_receipt().
Ending the lifecycle. revoke_generator() is terminal for a contract
ID and requires a newly reviewed contract for any later approval, but it is
not deletion. destroy_generator() removes every fitted generator held
under a contract, including its exact-row index, while retaining the
contract as a tombstone, the approval record marked destroyed with its
original approver, and the generation receipts as an audit trail. It is
idempotent and irreversible. It is an unlink, not a secure wipe: residual
copies on journalling filesystems, wear-levelled SSDs, snapshots, or
backups are outside the package's control, and the documentation says so.
Fail-closed generation. Generation runs the runtime privacy check and returns no usable output when it does not clear -- a verbatim source row reproduced, an infeasible k-anonymity target, or an exact-row check that could not be performed at all. The refusal names a durable receipt ID so the refusal itself is auditable.
Reproducibility scope. The fitted state records the engine, the DataGangeR package version, the generator schema version, the seed algorithm, and the canonical data-hash algorithm, and validates that block against the running runtime on every generation, failing closed on a mismatch. Effective seeds derive from the approved contract ID and the RNG kinds are pinned rather than inherited from the caller's session. Byte-for-byte reproduction is guaranteed only inside that recorded envelope; it is not a general promise across arbitrary version changes, and the R version is not part of the recorded block.
CLI. dataganger generator freeze|inspect|generate|revoke|destroy|status
covers the operator lifecycle. Approval remains an R or app action rather
than a CLI subcommand.
Gap. An agent that needed a new synthetic variation had to ask a human to reopen the real data, because there was no way for a process without data access to reach an approved generator.
What landed. A two-process route. `dataganger generator-broker --store
What the package claims, and what it does not. The package ships both
halves of the route and the handshake that verifies isolation. It does not
create the isolation; the host does. In-process package code cannot create
a boundary against a caller running as the same OS user, so the route
becomes available only when the running process proves at runtime that it
cannot read the private store as the current user. A successful read means
no boundary exists: the route reports itself unavailable, says why, and
generates nothing. status requires a configured broker, a capabilities
probe answered with the broker's own principal, a broker principal
different from the client's, a non-superuser client, and genuinely refused
real reads of the store marker and of generators/. Anything unexpected
reports unavailable, never available. Reference host configuration is
documented in the packaged agent skill's HOST-SETUP.md.
Symptom. The keyed exact-row fingerprint index, which lets generated output be screened for verbatim source rows, was derived in a form that did not reproduce portably across environments, so the screen could disagree with itself.
Fix. Fingerprints are computed over a canonical encoding and the
algorithm is recorded in the index as HMAC-SHA256-canonical-v1.
Migration. Any generator frozen before this fix carries the legacy
HMAC-SHA256 index and cannot be screened by the current runtime. Rather
than release output that was never actually checked, generation fails
closed with an exact_row_check_unavailable blocker stating that the
generator must be re-frozen and re-approved. There is no in-place upgrade:
the index derives from the real data, so re-freezing requires the source.
Re-run freeze_synthesis() with the same spec, roles, and limits, review
the new risk report, approve_generator() the new handle, and
destroy_generator() the old contract once the replacement is confirmed.
compare_synthetic() gained a utility component: the propensity-score
pMSE / S_pMSE diagnostic of Snoke, Raab, Nowok, Dibben & Slavkovic (2018),
the same formula synthpop::utility.gen() uses for its logistic-
regression method. S_pMSE near 1 means a model fit to distinguish
original from synthetic rows does no better than chance on the shared
predictor columns; higher values mean the two datasets are more
detectably different. Printed as its own "Utility" section, always
labeled as a utility measure, not a privacy measure -- it is not evidence
the data are safe to release.compare_synthetic() gained a disclosure component: role-aware
disclosure risk diagnostics powered by synthpop. When roles are
provided, identity disclosure is evaluated via replicated uniques
(synthpop::replicated.uniques()) across quasi-identifier keys,
measuring original uniques reproduced in synthetic data. When sensitive
target columns are identified in roles, attribute disclosure risk is
evaluated via the DiSCO measure (synthpop::disclosure()). Printed as
its own "Disclosure risk (synthpop)" section with an explicit reminder that
low replicated uniqueness does not guarantee immunity from
re-identification.roles identify a disclosure_role, synthesized columns are now
visited quasi/unclassified columns first (original order preserved),
sensitive columns last, so a sensitive column can never be a CART
predictor for a quasi or unclassified column. Output column order and
values are otherwise unaffected; behavior without roles, or without a
disclosure_role, is unchanged.synth_spec(..., synthpop_method = "parametric") is a new, advanced
(engine = "synthpop" only) option. It hands per-variable method
selection to synthpop's own type-driven defaults -- normrank for
numeric variables, logreg for two-level factors, polyreg for
unordered factors with more levels, polr for ordered factors -- instead
of running CART (nearest-neighbour donor imputation) on every column
regardless of type. Default is synthpop_method = "cart", which
reproduces every prior release's behavior exactly; nothing changes unless
this is set explicitly. Exposed in the app under Advanced settings ->
Output settings, visible only when the Engine dropdown is explicitly set
to synthpop (the "auto" engine still resolves to "cart" regardless).Symptom. generator_contract()'s compatibility block records the
package version, schema version, seed algorithm and data-hash algorithm
it was created under. Approving or generating from a contract created
under a different DataGangeR version -- the normal result of upgrading
the package while an older contract is still on file -- failed closed
with the same generic message used for genuine tampering (an edited
fingerprint, revision, risk report, or bounds), giving no indication that
this was an expected version boundary rather than an integrity failure.
Fix. A compatibility-envelope mismatch is never itself evidence of
tampering -- it is entirely a function of the installed package, not of
anything an attacker could forge to look legitimate -- so it now raises
its own dataganger_generator_store_error (not the tamper-specific
class), naming exactly which compatibility fields differ (most often just
package_version, recorded vs. running) and stating that the generator
must be re-frozen and re-approved under the current package version. Real
tampering checks (fingerprint, revision, risk report, bounds) are
unaffected and keep raising dataganger_generator_store_tamper_error.
README.md gained a section
introducing the capability and pointing at it.A portability release. Every item below is one defect: what went wrong, why, and what changed. The changes are limited to validated character date/time handling, test diagnostics, and portability corrections; privacy and export contracts are unchanged.
Symptom. A column such as "01/01/2020 01:15 PM" came back as
"01/01/2020 01:15" -- no period marker -- and every afternoon value
collapsed onto its morning hour, so 01:15 PM and 01:15 AM became the
same synthetic value. This appeared on some platform/locale pairs only,
which is why it surfaced on the CRAN check services and on macOS under
LC_TIME=en_GB while passing elsewhere.
Cause. %p is handed to the C library by both strptime() and
format(), and its meaning is locale- and OS-dependent: it can be
uppercase, lowercase, translated, or empty. Where the locale defines no
period marker, the 12-hour candidate format neither parsed nor
round-tripped, so detect_date_format() fell through to a 24-hour format
and discarded the marker together with the morning/afternoon distinction.
Fix. Detection and parsing read trailing ASCII AM/PM tokens
directly and apply the 12-to-24 hour correction in R. Output derives the
period from each synthesized hour and keeps the source column's upper- or
lowercase convention. When a %p candidate has no ASCII marker, the
package uses the active locale's native parser and formatter only if at
least 90 percent of non-missing source values round trip after surrounding
whitespace is trimmed. Unknown or mixed suffixes are rejected rather than
stripped or reinterpreted as 24-hour time. 24-hour and date-only formats
are unchanged.
Regression evidence. Helper and full-synthesis tests cover 12 AM,
12 PM, marker-case variants, an unknown FOO/BAR suffix fixture, a
C-locale translated-period rejection, and a conditional native-period test
when the host provides non-ASCII %p markers. The ASCII path is also
traced to prove it does not delegate %p to the C library.
Symptom. A permissive role-detection pattern could label month-name text as a date even when the active locale could not parse it. Synthesis then silently fell through to categorical resampling. Dotted and lowercase English forms could also be recognized by role detection but not preserve their format through the date path.
Cause. detect_roles() and synthesis made separate decisions: the
former used a broad regular expression, while the latter used locale-bound
strptime() candidates. detect_date_format() also selected the best
parsed candidate even when its round-trip fidelity was zero.
Fix. Role detection now calls the same validated format detector as synthesis, and every accepted format must round trip at least 90 percent of non-missing values. English abbreviated, full, dotted, and lowercase month forms use a locale-independent parser and renderer. A date role without a validated parser is now an error rather than a categorical fallback.
Regression evidence and limits. End-to-end seeded synthesis tests run
under LC_ALL=C for Jan, January, Jan., and lowercase English forms;
they assert role, parser selection, output shape, reparsing, and
determinism. Foreign month text, including accented examples, is supported
only when the active locale parses and formats the candidate with the
required fidelity. Otherwise it remains non-date text and is never silently
reinterpreted. This is not a claim of universal foreign-locale parsing.
Symptom. A check failure reported actual: FALSE / expected: TRUE and
nothing about the values that caused it, which is not enough to diagnose a
platform-specific defect from a check log.
Cause. Aggregate boolean assertions discarded the inspected values, so
remote check logs showed only actual: FALSE / expected: TRUE.
Fix and regression evidence. Package-wide any(grepl(...))
expectations now use matching expectations that print the inspected text.
Every remaining aggregate expect_true()/expect_false() supplies
diagnostic context that identifies the inspected or offending values. An
AST-based test scans every test file, reports file, line, and expression,
and rejects any aggregate boolean expectation without that context. Its
self-test inserts a deliberately opaque fixture and verifies the diagnostic.
LC_ALL=C; they are now ASCII x, with a regression that parses every
package R source file under LC_CTYPE=C.Generation now keeps its KPI row focused on output size, seed, duration, and exact matches. The k-anonymity outcome appears in a status card at the bottom of the page instead; an infeasible run gets a yellow warning with computed regenerate choices and a shortcut that opens the minimum-group- size control in Configure's Advanced settings. Export now leads with the bundle contents, omits the duplicate generation summary, and explains a missing k-anonymity protection using the run's actual k, QI columns, and smallest group size before asking for acknowledgment.
The minimum group size that drives k-anonymity is now a slider in the Configure step's Advanced settings, with live readouts of how many combinations and rows fall below the value you pick. It stays disabled until every column in the table above has both of its questions answered, because the setting has nothing to act on until then -- previously it could be dragged and silently did nothing. The slider unlocks on exactly the same condition as the Confirm button beside it, and dropped or passed-through columns need no answers for either.
Answering a column question no longer resets the other Advanced settings. The row count, engine, seed, column name handling, rare-category threshold and the checkboxes kept their values only until the next answer, which rebuilt the whole panel from the objective's presets. The suggested row count still tracks your answers, in the hint below the slider.
Configure step layout: the objective recap card is gone, since it restated a choice already made in the previous step and linked where the sidebar already goes. Advanced settings are split into privacy controls, shown first, and output settings, collapsed below them. The action legend is shorter, and the "pass through keeps the real values" caution now appears once in the legend rather than on every row -- it is unchanged in the Generate step, the export summary and the manifest.
Arriving at the Export step now opens a disclosure-risk brief: the minimum group size and which columns it covers, how many rows were blanked to reach it, how many synthetic rows reproduce a real record and how many of those expose a sensitive value, and the remaining privacy-check findings with their severity. The brief is informational only -- the existing export checks are unchanged, including the block on reproduced rows that expose a sensitive value. Its wording describes what each protection does rather than naming the technique.
Categorical output is now always plain character. Factors and
haven_labelled columns are normalized at synthesis input and are no
longer produced or returned by dataganger. A factor level declared with
zero rows is dropped: a category absent from the source stays absent from
the synthetic copy rather than being invented.
Logical columns now keep the logical type on both engines. Previously a
logical column came back as "TRUE"/"FALSE" text from the synthpop
engine because synthpop models it as a two-level factor.
The internal categorical resamplers no longer default to merging rare
values into .other. Every purpose preset already set merge_rare = FALSE,
so the old default matched no preset.
Demo and development synthesis now mask rare category labels instead of
merging them into .other, so every observed category keeps its own slot.
Every category observed in the source now appears in synthetic output for
demo and development purposes, while k-anonymity suppression no longer
introduces an (other) category.
Fix a stale merge_rare = TRUE fallback in the marginal engine. Every
purpose preset and every synth_*() helper defaults to FALSE, but a spec
that omitted the field entirely still merged rare labels into .other. A
spec built by synth_spec() always carries the field, so this was reachable
only through a hand-built spec, a recipe that omitted the key, or
synth_spec(merge_rare = NULL) -- assigning NULL drops the element.
Seeded synthesis now reproduces scrambled identifier columns exactly. The
pass that scrambles alphanumeric ID columns (simulation = "scramble",
the default for identifier-shaped columns) previously drew from ambient
randomness after the seeded step, so two runs with the same original data
and the same seed could produce different scrambled values even though
every other column matched. The scramble now runs under the spec's seed on
both engines, so the reproduction script in human/human.md delivers on
its same-seed, identical-output promise for every bundle.
Scrambled identifier columns (simulation = "scramble") no longer risk
handing back another record's identifier. Because scrambling permutes the
characters of the original value, one row's scramble could coincidentally
equal a different row's original ID (e.g. "T0010" scrambling to
"T0001"). Candidates are now checked against the full set of observed
values, so no original identifier from any row appears in the scrambled
output whenever the character space allows it.
Categorical resampling now guarantees every sampled category level appears at least once whenever the requested output has enough rows; this preserves level presence for downstream code paths, joins, and facets.
Capacity warnings for level-presence restoration are now combined into one warning per synthesis output while retaining affected columns and sizes.
Categorical and free-text columns gain a label_strategy setting and a
"Resample (rare levels masked)" action. It preserves each observed level's
slot and frequency while replacing rare labels with distinct neutral
placeholders, without merging categories into .other.
The Action dropdown on the Configure step now offers only the actions that
make sense for a column's data type, and names them for that type. Free
text gains an explicit choice between "Resample" and "Scramble"; numeric
and date columns no longer offer "Scramble", which would have run the
identifier scrambler over numbers; and alphanumeric IDs no longer offer a
resample. The single label "Synthesise" is replaced by the treatment it
actually applies -- "Resample" for categorical and free text, "Simulate"
for numeric and date, "Generate new" for postal codes. The stored action
vocabulary in roles$simulation is unchanged, so agent bundles and CLI
recipes are unaffected.
Postal-code columns lose their separate strategy dropdown: "Generate new" and "Resample" are actions, so they are now chosen in the Action dropdown alongside every other action. A postal column is configured with two controls instead of four. Country format remains its own control, as a parameter of the action rather than an alternative to it.
The Configure step's reference card is keyed by action rather than by data type, so the question it answers is "what will this choice do to my data" rather than "what type am I looking at".
An action override that a later type change makes meaningless is now discarded rather than left set, where it could be restored by a subsequent answer to question 1.
Fix silent data loss on the Configure step: a column you had explicitly
kept (Action = pass through or scramble) was reset to drop as soon as
you answered question 2 (Is it sensitive?). Sensitivity is not a keep/drop
decision, but the question-2 handler re-derived the column action anyway,
overwriting the choice. Because question 2 is mandatory, this affected
effectively every column detected as a direct identifier. Explicit column
actions are now recorded as a sticky override and are no longer clobbered
by either disclosure question.
Question 1 on the Configure step always offers "Yes, it identifies a person on its own". Previously, attesting at upload that the file held no direct identifiers removed that option, which left any column detected as a direct identifier displaying no answer at all, with no way to correct it. Answering "yes" still drops the column, by design, and now says so.
The Configure step stacks questions 1 and 2 in a single column to save horizontal space, and the drop notice is now driven by the effective column action rather than the disclosure role, so a direct identifier you chose to keep is no longer reported as removed.
New "Exact matches" tab in the data preview, shown when synthetic rows reproduce a real record verbatim. It lists one entry per row and column involved, with the row numbers in both tables and whether the column was marked sensitive in question 2. The Original and Synthetic previews now carry a row-number column so those references can be located.
Rows reproduced verbatim are tinted amber, or red when they expose a value
marked sensitive. Red rows block the browser export: the bundle cannot be
downloaded until a regeneration clears them, after which an explicit
acknowledgment is required and is recorded in agent/manifest.json as
exact_match_acknowledged. export_synthetic() gains a matching
exact_match_acknowledged argument, defaulting to FALSE.
Fix CRAN policy violation: rmarkdown::render() now passes
intermediates_dir = tempdir() so knitr writes intermediate files to the
R session's temporary directory rather than to the installed package
directory. The package directory is read-only on some CRAN platforms,
causing check failures on Debian r-devel and r-patched flavors.
A simpler column-role taxonomy, a de-identify-by-default policy for identifier columns, honest date/time synthesis, and a batch of Configure-page privacy and usability fixes.
generate (random format-valid values, zero source leakage)
and resample (observed values). Country format auto-detected from values
with explicit override. Postal config round-trips through recipe YAML.simulation = "scramble" — it is kept in the output but de-identified
(its letters and digits are reordered while delimiters and length are
preserved, so no original value survives) rather than silently removed.
Dropping a column is now an explicit action (simulation = "drop").
Recipes or scripts that relied on the old drop-by-default behaviour will
now see the column present (scrambled) unless they set simulation: drop
explicitly; regenerate affected bundles and update recipes as needed."5") or a run of
identical digits ("11"), so a plain integer ID column left its smallest
values in place — a re-identification leak. Such values are now replaced
with random digits of the same width instead, guaranteeing every scrambled
value differs from the original. Genuine alphanumeric IDs are still
reordered (their character multiset is preserved) as before."01/08/2020" or "14:30" were previously detected as
dates but fell through to generic categorical resampling — the original
values were reshuffled across rows with no date-range synthesis and no
coarsen_dates protection. They are now parsed with their own format,
synthesised through the date/datetime/time-of-day machinery, and
reformatted back to the source pattern. Character-stored dates also now
receive the same quasi disclosure default as native Date columns.kanon
attribute, manifest.json, and human/human.md carry suppressed_rows
and suppressed_row_frac, and dataganger inspect reports k-anonymity
status. A single below-k residual cell can cascade into suppressing much
more of a quasi-identifier column than its cell count implies; this is
existing, intended behaviour, now surfaced rather than hidden.stat risk styling now renders.dg_max_comparable_levels()), and the
former "pseudo identifier" type is merged into a single "alphanumeric ID"
identifier catch-all.CRAN resubmission changes requested on 2026-07-10.
'shiny' correctly in the
Description field.read_input() examples are now self-contained, executable, and no longer
use commented-out code.export_synthetic(), make_agent_bundle(), and
export_diagnostic_package() examples now write only to temporary paths
and no longer use \dontrun{}.tempfile() /
tempdir().One minimal export bundle, a Configure page with no silent defaults, and a package-wide audit pass: privacy fixes, an honest engine story, honest privacy wording, and a lighter dependency footprint.
name_strategy = "generic", generic renaming ran first and recorded the
name map before enforce_kanon() dropped direct identifiers and applied
suppression — so bundles could under-protect. Both engine paths now
enforce k-anonymity first. Bundles generated with generic names on
earlier versions may be under-protected; regenerate them.export_synthetic() uses the same roles-derived exclusion rule
as privacy_check_post().decimal_places() samples long
columns with a deterministic stride instead of sample(), removing an
RNG side effect.bslib::font_google() replaced with plain family strings served from
the packaged self-hosted files); the no-network source guard now also
scans inst/app/.synthetic_data.csv at the root,
human/ (human.md with the privacy report folded in, plus optional
comparison_report.html), and agent/ (recipe.yaml = combined
spec + roles, AGENT.md, manifest.json). data_dictionary.csv,
load_data.R, analysis.qmd, ai-readme.md, privacy_report.txt, and
the separate spec.yaml/roles.yaml are gone. CLI synthesize gains
--recipe <recipe.yaml>; --spec/--roles remain supported. compact
and include_dictionary are deprecated no-ops.export_synthetic() honors its code_readiness argument: when
supplied, the bundle gains agent/code_readiness_report.json;
make_agent_bundle() computes it automatically, so every agent bundle
now ships the structural-compatibility report.AGENT.md in every bundle and
using-dataganger-bundles) rewritten to the minimal bundle contract."auto" is a real engine alias in synth_spec(), synthesize_data(),
the CLI (--engine <auto|internal|synthpop>), and spec YAML. An explicit
"auto" behaves exactly like leaving the engine unset: the engine is
derived from the objective and dataganger.disable_synthpop is
respected.engine_required spec field is retired; spec printing
reports the explicit engine or auto (derived from objective).synthpop is unavailable, the upload attestation recommends installing
it for correlation-aware synthesis.user_identifies / user_sensitive) so CLI synthesis is
unaffected.income; flagged columns get a "potential identifier" pill and
semantically-coloured actions.detect_roles() documents the two-axis columns; assorted roxygen and
vignette corrections; the startup message is reduced to version +
run_app().purrr, tidyr, and vctrs dropped from Imports (unused); plotly
moved to Suggests with an install gate in the Compare module.check_code_readiness() now documents and reports that
haven_labelled → character is the expected round-trip for now.Privacy gating, UI/CLI parity, an agent skill, and a provable no-network guarantee.
spec.yaml and roles.yaml, and CLI
synthesize --roles can reuse the full role matrix so UI and CLI runs
reproduce byte-identical output with the same seed.none / combination. The two questions are
framed as the remaining risks after direct identifiers: linkage (combination)
and sensitivity.SKILL.md plus dataganger skill [--out <file>]
so an AI can drive the package to generate synthetic data without ever
reading the real data; fixed ai-readme.md so dropped columns are not listed
as NA (NA).report_issue() prints a copy-paste
GitHub issue instead of opening a browser (the Shiny button shows a copyable
modal). A shipped runtime trap test and source guard prove the package makes
no network calls, and a Linux unshare -rn CI job runs the suite with no
network at all.vignette("privacy-and-ai-workflow") documents the privacy gating ladder,
the two ways to use the package with AI, and the no-network guarantee.Configure redesign around two intrinsic privacy questions.
identifies: none / combination /
direct) and is it sensitive (sensitive) — and derives the treatment
rather than asking the user to pick it. The two questions are shown
prominently above the per-column table.NA bins.synth_spec() documentation for the current settings surface.export_synthetic(compact = ) supports two bundle variants: the compact app
download and the full CLI / agent bundle.Generation, comparison, and export clarity pass:
report_issue() helper plus an in-app Report a problem button open
a pre-filled GitHub issue with environment details, without sending anything
automatically.synthpop (auto))
after generation, instead of always showing auto.:::
self-reference (clearing an R CMD check NOTE); spelling WORDLIST expanded.Configure page clarity pass:
127, not
127.00) by detecting whole-valued numerics, not just R integer storage.... (recommended)), drops the
redundant recommended_role column, and adds a per-row info tooltip with a
plain-English reason and the storage type.+/- affordance.pkgload removed from Suggests (uses .__DEVTOOLS__ namespace check directly).cran-comments.md refreshed with accurate 0/0/2 NOTE explanations.pkgdown site generation via GitHub Actions (_pkgdown.yml + workflows).docs/superpowers/, todo.md) migrated out of the package root.detect_roles() is now conservative: it only auto-assigns a disclosure role
when confident (a clear direct identifier, or a known-sensitive column name).
All other columns are left unselected rather than defaulted to
quasi-identifier. This fixes the root cause of the 100%-NA synthetic output:
measures, counts, dates, and low-cardinality categoricals are no longer
silently treated as quasi-identifiers.None is a valid explicit choice; empty is not.max_suppress_frac feasibility backstop is retained as defense-in-depth.disclosure_roles: mapping (column -> role) so
disclosure decisions are reproducible from the command line.Bug fix — CUSUM hang (Bug 5): Synthesis no longer hangs on datasets with
character-stored date columns (e.g. "Jun 8, 2019") or other high-cardinality
character columns. The root cause was two-fold: (1) character date strings were
classified "unknown" and passed to synthpop as a 2000+-level factor, causing
CART to enumerate billions of split candidates; (2) even moderate-cardinality
character columns (>20 distinct values) used as CART predictors trigger the
same 2^(k-1) blowup at k=34. Fix: detect_roles() now detects date strings
(ISO, "Mon DD YYYY", MM/DD/YY) via regex and classifies them as "date"; and a
new synthpop_bridge_cols() function excludes any character column with >20
distinct values from synthpop's CART, synthesizes it independently via the
marginal engine, and stitches it back into the output in the original column
order. Generation on the CUSUM test file (41 k rows, 14 columns) now completes
in under 10 seconds at both 50-row and 5000-row target sizes.
P1 — Configure busy indicator: the Upload page now shows a "Profiling data…" / "Detecting column roles…" progress bar while the app analyses the uploaded file, so users know the app is working rather than frozen.
P2 — Row count first: the Row count (n) input is now the first item in the Configure advanced-settings panel.
P3 — Role-reactive row suggestion: suggest_min_rows() gains a data
parameter; when called with data and roles, it recomputes the coverage
estimate over only the columns that are still being synthesized. Dropping an
ID or excluded column now immediately lowers the suggested row count on the
Configure page.
P4 — Case IDs render as character: columns detected as ID candidates are coerced to character in the data-panel preview, so a numeric case ID displays as "1078541" instead of "1,078,541.00".
P5 — Column summary stats: the Configure page now shows a per-column summary section below the synthesis settings. Continuous columns get a min / Q1 / median / Q3 / max / mean / SD table; categorical columns get a top-5 frequency table with counts and percentages.
P6 — Generation progress bar + timer: while synthesis is running, the
Generation page displays a MM:SS elapsed-time counter and an animated
progress bar. Both update every second.
Cancellable background synthesis: the Shiny Generation step now runs the
synthesize -> compare -> privacy pipeline in a background process (via
callr), so the app stays responsive and a Cancel button can stop a long or
stuck run. Falls back to synchronous in-process generation when callr is
unavailable or the package is dev-loaded. The dataganger.synthesis_async
option forces the deterministic synchronous path for tests and CI.
Coverage-based row-count suggestion: profile_data() now carries
cross-column coverage (distinct joint combinations of categorical columns
plus the largest per-column level count), and the new suggest_min_rows()
function turns that into a sufficient synthetic row count - capped at 5000,
floored at the largest level count, and never above the original. The
Configuration row-count slider pre-fills with the suggestion, shows an
inline hint, and warns when set below the coverage floor. The Upload step
shows a coverage-summary card.
Diagnostics for long-running synthesis: dg_log / dg_timeit emit
per-phase progress to the R console when options(dataganger.verbose = TRUE), and check_cancel() polls options(dataganger.cancel) at column
boundaries for cooperative cancellation. .onAttach() prints a startup
hint with the package version and how to launch the app and CLI.
Free-text detection now head-samples to 1000 rows, bounding a hot path that could slow the transition into Configure on wide or long-string data.
synthesize_marginal() trusts the detected role instead of recomputing the
free-text heuristic, removing a redundant pass over character columns.
The Comparison step no longer shows a stale full table on first transition: the first comparable variable renders correctly without needing a click.
The Configuration step shows inline help for each disclosure role (None, Direct identifier, Quasi-identifier, Sensitive) with a short example.
callr and pkgload added to Suggests.
The Column Roles step now shows a non-blocking notice when the uploaded data looks like an aggregated counts table rather than individual records, since disclosure control assumes individual-level microdata.
New option dataganger.disable_synthpop: set
options(dataganger.disable_synthpop = TRUE) to steer objective-derived
synthesis onto the internal engine even when synthpop is installed. Intended
for environments where a synthpop synthesis is undesirable or can hang
unattended (for example continuous integration). An explicit
engine = "synthpop" request is still honoured.
Shiny app interface refinements: the Objective step shows each purpose's details on selection and uses consistent "more bars = stronger" meters, including an Anonymity meter for resistance to re-identification. Each synthesis engine (auto, internal, synthpop) now has a plain-language explainer. The Generation step shows a configuration recap (including advanced settings) with an "Adjust settings" shortcut back to Configuration. The Comparison step has a wrapping variable grid and a clearer original vs. synthetic explainer. The Export step downloads a single bundle (synthetic data as CSV, documentation, comparison report, and an analysis notebook), with an optional save-to-folder for sessions run locally.
Every bundle now includes analysis.qmd, a Quarto report with runnable R
code and reference Python code to read both the original and synthetic data
and compare them (summary statistics, distribution plots, and DataGangeR's
fidelity metrics).
The Shiny synthesis spec exposes an engine selector (auto / internal / synthpop) with an inline note on whether synthpop is installed.
Ship an agent skill (inst/skills/using-dataganger-bundles/) describing how
AI agents should consume a bundle: never access the real data, ask the human
for a go-ahead before touching the synthetic data, and where to save work.
Fixed interpolate() so bundle helper files (e.g. load_data.R) render
their templates instead of shipping literal placeholders.
DataGangeR now routes relationship-preserving objectives to the optional synthpop engine when installed. Please cite: Nowok B, Raab GM, Dibben C (2016). "synthpop: Bespoke Creation of Synthetic Data in R." Journal of Statistical Software, 74(11), 1-26. doi:10.18637/jss.v074.i11
Initial CRAN-ready scaffold.