TL;DR; This blog post ends with minimal example of creating Plone 5.2 content with Python 3 compatible Transmogrifier pipeline with command line execution.
Years ago, I forked the famous Plone content migration tool Transmogrifier into a Plone independent and Python 3 compatible version, but never released the fork to avoid maintenance burden. Unfortunately, I was informed that my old examples of using my transmogrifier fork with Plone no longer worked, so I had to review the situation.
The resolution: I found that I had changed some of the built-in reusable blueprints after the post, I updated the old post, fixed a compatibility issue related to updates in Zope Component Architecture dependencies, and tested the results with the latest Plone 5.2 on Python 3.
Transmogrifying RSS into Plone
So, here goes a minimal example for creating Plone 5.2 content with Python 3 Transmogrifier pipeline using my fork:
At first ./buildout.cfg for the Plone instance:
[buildout]extends = http://dist.plone.org/release/5-latest/versions.cfgparts = instance plonesiteversions = versionsextensions = mr.developersources = sourcesauto-checkout = *[sources]transmogrifier = git https://github.com/collective/transmogrifier[instance]recipe = plone.recipe.zope2instanceeggs = Plone transmogrifieruser = admin:admin[plonesite]recipe = collective.recipe.plonesitesite-id = Ploneinstance = instanceThen buildout must be run to create the instance with a Plone site:
$ buildoutNext the transmogrifier ./pipeline.cfg must be created to define the pipeline:
[transmogrifier]pipeline = from_rss prepare create patch commit[from_rss]blueprint = transmogrifier.frommodules = feedparserexpression = python:modules['feedparser'].parse(options['url']).get('entries', [])url = http://rss.slashdot.org/Slashdot/slashdot[prepare]blueprint = transmogrifier.setportal_type = string:Documentid = python:Nonetext = path:item/summary_container = python:context.get('slashdot') or modules['plone.api'].content.create(container=context, type='Folder', id='slashdot')[create]blueprint = transmogrifier.setmodules = plone.apiobject = python:modules['plone.api'].content.create(container=item.pop('_container'), type='Document', **item)[patch]blueprint = transmogrifier.transformmodules = plone.app.textfieldpatch = python:setattr(item['object'], 'text', modules['plone.app.textfield'].value.RichTextValue(item['object'].text, 'text/html', 'text/x-html-safe'))[commit]blueprint = transmogrifier.finallymodules = transactioncommit = modules['transaction'].commit()Finally, the execution of transmogrifier with Plone site as its context (remember that this version of transmogrifier also works outside Plone ecosystem, but for a convenience transmogrify-script also supports calling with instance run):
$ bin/instance -OPlone run bin/transmogrify pipeline.cfg --context=zope.component.hooks.getSiteThis example should result with the latest Slashdot posts in a Plone site. And, because this example is not perfect, running this again would create duplicates.
Transmogrifying JSON files into Plone
There’s never enough simple tutorials on how to build your own Transmogrifier pipelines from scratch. Especially now, when many old pipeline packages have not been ported to Python 3 yet.
In this example we configure a buildout with local custom Transmogrifier blueprints in python and use them to do minimal import from a JSON export generated using collective.jsonify, which is a one of many legacy ways to generate intermediate export. (That said, it might be good to know, that nowadays trivial migrations could be done with just Plone REST API and a little shell scripting.)
At first, we will define a ./buildout.cfg that expects a local directory ./local to contain a Python module ./local/custom and include ZCML configuration from ./local/custom/configure.zcml:
[buildout]extends = http://dist.plone.org/release/5-latest/versions.cfgparts = instance plonesiteversions = versionsextensions = mr.developersources = sourcesauto-checkout = *[sources]transmogrifier = git https://github.com/collective/transmogrifier[instance]recipe = plone.recipe.zope2instanceeggs = Plone transmogrifier plone.restapiuser = admin:adminextra-paths = localzcml = custom[plonesite]recipe = collective.recipe.plonesitesite-id = Ploneinstance = instanceBefore running buildout we ensure a proper local Python module structure with:
$ mkdir-p local/custom$ touch local/custom/__init__.py$ echo'<configure xmlns="http://namespaces.zope.org/zope" />'> local/custom/__init__.pyOnly then we run buildout as usually:
$ buildoutNow, let’s populate our custom module with a Python module ./local/custom/blueprints.py defining a couple of custom blueprints:
# -*- coding: utf-8 -*-fromtransmogrifier.blueprintsimportBlueprintimportjsonimportpathlibclassGlob(Blueprint):"""Produce JSON items from files matching globbing from option `glob`."""def__iter__(self):foriteminself.previous:yielditemforpinpathlib.Path(".").glob(self.options["glob"]):withopen(p, encoding="utf-8") asfp:yieldjson.load(fp)classFolders(Blueprint):"""Minimal Folder item producer to ensure that items have containers."""def__iter__(self):context=self.transmogrifier.contextforiteminself.previous:parts= (item.get('_path') or'').strip('/').split('/')[:-1]path=''forpartinparts:path+='/'+parttry:context.restrictedTraverse(path)exceptKeyError:yield {"_path": path,"_type": "Folder","id": part }yielditemAnd complete ZCML configuration at ./local/custom/configure.zcml with matching blueprint registrations:
<configurexmlns="http://namespaces.zope.org/zope"xmlns:transmogrifier="http://namespaces.plone.org/transmogrifier"><includepackage="transmogrifier"file="meta.zcml"/><transmogrifier:blueprintcomponent=".blueprints.Glob"name="custom.glob"/><transmogrifier:blueprintcomponent=".blueprints.Folders"name="custom.folders"/></configure>Now, by using these two new blueprints and minimal content creating pipeline parts based on built-in expression blueprints, it is possible to:
- generate new pipeline items from exported JSON files
- inject folder items into pipeline to ensure that containers are created before items (because we cannot quarentee any order from the export)
- create minimal Folder and Document objects with plone.api.
[transmogrifier]pipeline = generate_from_json generate_containers set_container create_folder create_document commit[generate_from_json]blueprint = custom.globglob = data/**/*.json[generate_containers]blueprint = custom.folders[set_container]blueprint = transmogrifier.set_container = python:context.restrictedTraverse(item["_path"].rsplit("/", 1)[0])[create_folder]blueprint = transmogrifier.setcondition = python:item.get("_type") =="Folder"modules = plone.api_object = python:modules["plone.api"].content.get(item["_path"]) or modules["plone.api"].content.create(container=item["_container"], type="Folder", id=item["id"])[create_document]blueprint = transmogrifier.setcondition = python:item.get("_type") =="Document"modules = plone.api plone.app.textfield_object = python:modules["plone.api"].content.get(item["_path"]) or modules["plone.api"].content.create(container=item["_container"], type="Document", id=item["id"], title=item["title"], text=modules['plone.app.textfield'].value.RichTextValue(item["text"], 'text/html', 'text/x-html-safe'))[commit]blueprint = transmogrifier.finallymodules = transactioncommit = modules['transaction'].commit()Finally, the pipeline can be run and content imported with:
$ bin/instance -OPlone run bin/transmogrify pipeline.cfg --context=zope.component.hooks.getSiteObviously, in a real migration, the pipeline parts [create_folder] and [create_document] should be implemented in Python to properly populate all metadata fields, handle possible exceptions, etc, but consider that as homework.
If this post raised more questions than gave answers, please, feel free to ask more at: https://github.com/collective/transmogrifier/issues.