``rattail.importing``

=====================

.. automodule:: rattail.importing

There are a handful of importers and import handlers made available within this

namespace, i.e. not just the base classes but some variations thereof etc.  You

may import each of these directly from this namespace, e.g.::

   from rattail.importing import Importer

However it's more typical to do this instead::

   from rattail import importing

That way you can reference ``importing.Importer`` as well as

``importing.model.ProductImporter`` etc.  The full list of what's available in

this ``rattail.importing`` namespace follows.

Importers

---------

 * :class:`rattail.importing.importers.Importer`

 * :class:`rattail.importing.importers.FromQuery`

 * :class:`rattail.importing.importers.BulkImporter`

 * :class:`rattail.importing.sqlalchemy.FromSQLAlchemy`

 * :class:`rattail.importing.sqlalchemy.ToSQLAlchemy`

 * :class:`rattail.importing.postgresql.BulkToPostgreSQL`

Import Handlers

---------------

 * :class:`rattail.importing.handlers.ImportHandler`

 * :class:`rattail.importing.handlers.BulkImportHandler`

 * :class:`rattail.importing.handlers.FromSQLAlchemyHandler`

 * :class:`rattail.importing.handlers.ToSQLAlchemyHandler`

 * :class:`rattail.importing.rattail.FromRattailHandler`

 * :class:`rattail.importing.rattail.ToRattailHandler`

Rattail Model Importers

-----------------------

This is a little different but worth a mention.  If you do::

Data Importers

==============

A frequent need when integrating systems is to import (or export, depending on

framework for this, which offers the following benefits:

 * "dry run" mode to check things out before committing changes

 * "warnings" mode which sends email with data diffs, e.g. when you expect no changes

 * adjustable "batch size" for grouping changes when submitting to local system

 * full command line support for above, plus "max" changes to apply, show progress, etc.

 * core code is optimized to run quickly, e.g. by fetching all data up-front

 * new importers may be created simply / cleanly / according to existing patterns

 * new importers may extend / replace core functionality as needed

The rest of this document aims to explain the concepts and patterns involved

with the Rattail importer framework.

"Importer" vs. "DataSync"

-------------------------

Perhaps the first thing to clear up, is that while Rattail also has a

"datasync" framework, tasked with keeping systems in sync in real-time, the

"importer" framework is tasked with a "full" sync of two systems.  In other

words "datasync" normally deals with only one (e.g. changed) "host" object at a

time, and will update the "local" system accordingly, whereas an "importer"

will examine *all* host objects and update local system accordingly.  Also,

datasync normally runs as a proper daemon, whereas an importer will normally

run either as a cron job or in response to user request via command line or web

UI, etc.

.. todo::

   Write datasync docs / link here.

To make things even more confusing, datasync can leverage an import handler /

importer(s) where possible so that the same logic is executed for both

"real-time sync" and "full sync" modes.

"Host" vs. "Local" Systems

strictly necessary; for instance you might need an importer to push new

customer email addresses to an online mailing list, and it may have to use a

web API which only supports one add per call.  In other words you have only one

"data model" to update, so you don't need a handler to manage multiple

importers, and the web API doesn't support the commit/rollback approach because

each change submitted, is committed at once.  However the suggested approach is

to stick with established patterns and use a handler; various other parts of

the Rattail framework (command line, datasync) will expect one.

Making a new Importer

---------------------

Okay then, you must be serious if you made it this far...

First step of course will be to identify the "host" and "local" systems for

your particular scenario.  For the sake of a simple example here we'll assume

you wish to import product data from your "host" point of sale system (named

"MyPOS" within these docs) to your "local" Rattail system.

Note also that to make a new importer, you must have already started a project

based on Rattail; this doc will not explain that process.  The examples which

follow assume this project is named 'myapp'.

File / Module Structure

^^^^^^^^^^^^^^^^^^^^^^^

With the host and local systems identified, you can now start writing

code...but where to put it?  Assuming you already have a Rattail-based project

with package named 'myapp' and assuming you were adding a POS->Rattail

importer, the suggestion would be to add the following files to your project:

.. code-block:: none

   myapp/

      __init__.py

      importing/

         __init__.py

         model.py

         mypos.py

This is just a suggestion really, although it is the author's personal

convention which has served him well.  Another typical scenario might be where

you wish to "export" data from Rattail->POS, in which case you might do

something like this instead: