In light of the recent plone4.csrffixes security fix, I’d like to share some of our experiences in debugging and fixing CSRF protection false positives. Because plone.protects approach for automatic CSRF protection is pretty comprehensive (which is good), it can result in cases where there’s false positives – a dialog that is shown to the user asking them to confirm their intent (to prevent the request forgery), even though no actual CSRF attack has occurred.

We’ve been using the automatic CSRF protection from plone.protect 3.x with Plone 4 for a little more than half a year now, before it was officially supported. We therefore hit quite a few situations where we had to debug false positives caused by a write-on-read, both the ones in stock Plone 4 (which now have been addressed), and ones caused in our own add-ons.

Particularly because of the recently introduced HTTP_REFERER check you should rarely ever hit those false positives any more, but if you do, here’s some techniques for debugging and fixing them.

How plone.protect’s auto CSRF protection works

CSRF protection (automatic or manual) in Plone is done via plone.protect. Plone 4 used to pin plone.protect == 2.x, which put a basic framework for manual CSRF protection in place. plone.protect >= 3.0, which was targeted at Plone 5, then introduced automatic CSRF protection.

The automatic CSRF protection works by requiring requests to include a CSRF token that has been issued by the application.

The basic logic for checking a request is as follows:

If the user is authenticated AND the request caused a DB write, require a valid CSRF token.

If no valid CSRF token can be found, the transaction is aborted and the user is redirected to a confirmation page where they have to confirm their intent by clicking a button and submit the original request again, this time including a valid token. (That confirmation page is only displayed for GET requests with responses of type text/html, for other requests the transaction will simply be aborted).

There’s a few exceptions to this logic for handling special cases, but that’s the gist of it. For all the gory details, see the ProtectTransform._check() method in the plone.protect.auto module.

Now, in order to ensure that legitimate requests that modify the DB contain the required CSRF token, plone.protectapplies a transform to (most) responses of content type text/html. This transform inserts a hidden field with a CSRF token into any <form> contained in that response. This means that for most cases where you modify the DB by submitting forms via a POST request, the necessary CSRF token will automatically have been included for you.

What plone4.csrffixes does

The plone4.csrffixes package addresses the ZMI vulnerabilities mentioned in the security advisory by pulling in plone.protect >= 3.x instead of a default Plone 4’s plone.protect == 2.x. This will enable the automatic CSRF protection mentioned above for your entire Plone site and the Zope Application Root.

The actual code in the plone4.csrffixes package just contains necessary changes to avoid false positives with automatic CSRF protection in Plone 4:

• Some JavaScript that adds the necessary authenticator tokens to AJAX requests
• Additional transforms that whitelist known writes-on-read
• A backstop that whitelists requests with a HTTP_REFERER or HTTP_ORIGIN equal to the site’s URL
• A transform that rewrites links in the ZMI to include the authenticator token
• A transform that adds authenticator tokens to links that cause known writes-on-read

So to summarize, the actual automatic CSRF protection happens in plone.protect >= 3.0, while the code in plone4.csrffixes is just Plone 4 compatibility glue to avoid false positives.

How to debug CSRF false positives

Triggering the CSRF protection intentionally

During development and/or debugging, it can be helpful to be able to trigger the CSRF protection dialog intentionally. For example in order to

• See what your users see if they do get a CSRF protection dialog
• Style that dialog and test potential overrides to the template
• Test your own CSRF protection debugging tools
• Test logging of CSRF incidents

For simple cases, you can simply call the @@confirm-action view and supply it with a value for original_url, for example by visiting http://localhost:8080/Plone/@@confirm-action?original_url=foo

But there may be situations where you really want to put the automatic CSRF protection through its paces during development or debugging. For that purpose, a simple view does the trick:

Code: trigger.py (gist)

Simply call that view by visiting http://localhost:8080/Plone/@@trigger-csrf directly (i.e. not via a link from your actual Plone site, otherwise the HTTP referrer check will whitelist the write).

Inspecting _registered_objects

If you’re developing an add-on, and you suddenly get hit by the CSRF protection dialog, indicating that there is an unexpected write-on-read, it can be quite tricky to figure out where that write is happening.

In order to determine the root cause, you first need to understand one detail about how plone.protect’s automatic CSRF protection works: The ProtectTransform in plone.protect.auto looks at conn._registered_objects to determine whether a database write occured in the current transaction or not. You can basically think of _registered_objectsas the list of dirty objects in a ZODB connection: The first time an object is modified, it gets added to _registered_objects.

This allows you to easily get a hold of the offending objects (those that have been written to) by setting a PDB breakpoint in plone.protect.auto and then inspect self._registered_objects() (ignoring the ones listed in safe_oids).

Depending on the modified object(s), that may or may not give you a clue to which part of the code base actually modified that object. In my experience, more often than not, you’re just sitting there and staring at a __dict__ dump or some annotations, not even knowing what on the object got changed, let alone where from.

For situations like these, there’s two approaches that helped me in the past.

Tracking down references to the object

Using Python’s garbage collector, you can get a list of references to any value. Simply call gc.get_referrers(obj) and you’ll get a list of referrers:

123

(Pdb) import gc
(Pdb) obj = self._registered_objects()[0]
(Pdb) pp gc.get_referrers(obj)

Note that this list will also include your local scope and any references you created while debugging, for example while in PDB.

This approach is usually a long shot, but it might for example help you figure out that some persistent data structure is referenced from some annotations, and you then can follow that up by tracking down references to those annotations, hopefully finding their context.

But still, this doesn’t really lead you to the code causing the write in a reliable, systematic way, that’s why I often go straight for the second approach.

Tracing object registrations

In order to capture writes to persistent objects, I wrote a helper that intercepts calls to a ZODB Connection’s register() method using a call trace function, and dumps a stack trace that should include the strack frame where the DB write was caused.

Code: trace_register.py (gist)

Since this will dump a stack trace every time an object gets first written to, I implemented this helper as a context manager, so it can be used with as narrow a scope as possible (the context manager removes the call trace function in its __exit__() method).

So with the @@trigger-csrf view from above, you’d use it like this:

12

withTraceObjectRegistrations(tb_limit=5):self._do_write()

Visiting that view will then dump a stack trace like the following:

123456789101112131415

==================================================================DBwriteto<PloneSiteatPlone>(0x11)from"trigger.py",line16==================================================================File".../ZPublisher/Publish.py",line138,inpublishrequest,bind=1)File".../ZPublisher/mapply.py",line77,inmapplyifdebugisnotNone:returndebug(object,args,context)File".../ZPublisher/Publish.py",line48,incall_objectresult=apply(object,args)# Type s<cr> to step into published object.File"trigger.py",line11,in__call__self._do_write()File"trigger.py",line16,in_do_writeself.context.myattr='foo'2015-10-1117:02:17INFOplone.protectabortingtransactionduetonoCSRFprotectiononurlhttp://localhost:8080/Plone/@@trigger-csrf

This should show you the exact location in your code where the DB write was caused.

Notes:

• This helper is intended for DEBUGGING, not for use in production!
• This helper will dump stack traces for all calls to register(), not just the ones triggering the confirmation dialog

How to fix your code

Don’t cause writes in GET requests

If you can at all avoid it, don’t cause DB writes in your GET requests.

While plone.protect tries to insert CSRF tokens automatically for you in forms, it can’t really do so for plain <a href="..."> links without making all your URLs look hideous.

With GET requests there’s also the possibility of leaking the CSRF token, for example through log files or the HTTP_REFERER (unless your site strictly enforces HTTPS).

OWASP has the following to say about Disclosure of Token in URL:

The ideal solution is to only include the CSRF token in POST requests and modify server-side actions that have state changing affect to only respond to POST requests. This is in fact what the RFC 2616 requires for GET requests. If sensitive server-side actions are guaranteed to only ever respond to POST requests, then there is no need to include the token in GET requests.

Provide CSRF tokens for the GET requests that need them

If you still have a GET request that needs to write, you’ll need to make sure it includes an authenticator token. For that you can use the addTokenToUrl() helper function from plone.protect:

123

fromplone.protect.utilsimportaddTokenToUrlurl=addTokenToUrl(url)

Avoiding writes on read

Lazy initialization of persistent data structures

One common source of a write-on-read is lazy initialization of persistent data structures. Consider the following code:

1234567

CONFIG_KEY='foo-config'defget_config(self):annotations=IAnnotations(self.context)ifnotCONFIG_KEYinannotations:annotations[CONFIG_KEY]=PersistentDict()returnannotations[CONFIG_KEY]

Because annotations[CONFIG_KEY] is lazily initialized the first time it’s accessed, this means that on the first request that attempts to just read the config, you will cause a DB write.

While code like this is generally easier to write, it should be avoided because not only could it trigger the automatic CSRF protection, it can also negatively impact performance and lead to write conflicts.

Some alternatives are:

• Account for the possibility of the value not existing yet in places where you attempt to access it, and handle that case accordingly.
• Ensure that persistent data structures like these are being initialized upon object creation, and write the necessary upgrade steps to initialize them on existing objects

Intended writes-on-read

There are also some cases where you actually may want to write some data during an operation that looks like it should be read-only. One case that comes to mind is logging access to resources:

Say you need to keep a journal for your file-like resources that tracks username and timestamp for every download of that file. Unless you store that information outside the ZODB (e.g. on a file system log, in which case you’ll lose transactionality), you will inevitably cause a DB write on every download.

If for some reason you can’t make these actions use POST requests, and can’t include authenticator tokens in the GET request, you’ll have to whitelist those writes.

Whitelisting known writes

If you do find yourself in a situation where you still have a write-on-read you can’t eliminate, you can whitelist that write in one of two ways:

Whitelisting a persistent object that’s being written to

plone.protect includes a function safeWrite(obj, request=None) that whitelists a specific persistent object as a safe write, by adding it to a list of whitelisted objects, and then skips those when checking _registered_objects() for modified objects.

123

fromplone.protect.autoimportsafeWritesafeWrite(myobj,request)

This is pretty much the approach we came up with while using plone.protect == 3.x with Plone 4 when it wasn’t officially supported yet. It works well, and allows you to precisely whitelist specific objects, as opposed to disabling CSRF protection completely for the entire request.

Completely disabling CSRF protection for certain requests

If nothing else works, you have the ability to completely disable automatic CSRF protection for an entire request by having it provide the IDisableCSRFProtection interface:

123

fromplone.protect.interfacesimportIDisableCSRFProtectionfromzope.interfaceimportalsoProvidesalsoProvides(request,IDisableCSRFProtection)

However, this should only be used as a last resort, since it makes the request susceptible to CSRF attacks, and requires you to perform any CSRF protection yourself.

You may be tempted to use this in your functional or integration tests – but even in tests, I’d recommend using IDisableCSRFProtection sparingly, because your tests are an excellent way to discover writes-on-read you didn’t know about, and allow to fix the issue (a false positive) before it reaches production, and potentially scares or annoys customers.

Closing thoughts

Overall I really appreciate the backport of plone.protects automatic CSRF protection to Plone 4. This is something that might not be obvious from the security advisory: Not only have some ZMI issues been fixed, but we also get full, automatic CSRF protection for all our forms, custom or not. In my opinion that’s a huge win, even though there may be a couple integration issues until people get used to dealing with the automatic CSRF protection.

The following text is in German, since we're announcing a regional user group meeting in Düsseldorf, Germany.

Ankündigung

Das nächste Python Meeting Düsseldorf findet an folgendem Termin statt:

Mittwoch, 21.10.2015, 18:00 Uhr
Raum 1, 2.OG im Bürgerhaus Stadtteilzentrum Bilk
Düsseldorfer Arcaden, Bachstr. 145, 40217 Düsseldorf

Neuigkeiten

Bereits angemeldete Vorträge

Matthias Endler
       "Writing a fuzzy receipt parser in Python using Tesseract"

Marc-Andre Lemburg
       "Untwisting Mersenne Twister: Python's Zufallszahlengenerator"
       "Portierung von mxDateTime auf Python 3"

Weitere Vorträge können gerne noch angemeldet werden. Bei Interesse, bitte unter info@pyddf.de melden.

Startzeit und Ort

Wir treffen uns um 18:00 Uhr im Bürgerhaus in den Düsseldorfer Arcaden.

Das Bürgerhaus teilt sich den Eingang mit dem Schwimmbad und befindet sich an der Seite der Tiefgarageneinfahrt der Düsseldorfer Arcaden.

Über dem Eingang steht ein großes “Schwimm’in Bilk” Logo. Hinter der Tür direkt links zu den zwei Aufzügen, dann in den 2. Stock hochfahren. Der Eingang zum Raum 1 liegt direkt links, wenn man aus dem Aufzug kommt.

>>> Eingang in Google Street View

Einleitung

Das Python Meeting Düsseldorf ist eine regelmäßige Veranstaltung in Düsseldorf, die sich an Python Begeisterte aus der Region wendet.

Veranstaltet wird das Meeting von der eGenix.com GmbH, Langenfeld, in Zusammenarbeit mit Clark Consulting & Research, Düsseldorf:

Programm

Das Python Meeting Düsseldorf nutzt eine Mischung aus Open Space und Lightning Talks, wobei die Gewitter bei uns auch schon mal 20 Minuten dauern können :-)

Lightning Talk Anmeldung bitte formlos per EMail an info@pyddf.de

Kostenbeteiligung

Das Python Meeting Düsseldorf wird von Python Nutzern für Python Nutzer veranstaltet.

Da Tagungsraum, Beamer, Internet und Getränke Kosten produzieren, bitten wir die Teilnehmer um einen Beitrag in Höhe von EUR 10,00 inkl. 19% Mwst. Schüler und Studenten zahlen EUR 5,00 inkl. 19% Mwst.

Wir möchten alle Teilnehmer bitten, den Betrag in bar mitzubringen.

Anmeldung

Da wir nur für ca. 20 Personen Sitzplätze haben, möchten wir bitten, sich per EMail anzumelden. Damit wird keine Verpflichtung eingegangen. Es erleichtert uns allerdings die Planung.

Meeting Anmeldung bitte formlos per EMail an info@pyddf.de

Weitere Informationen

Weitere Informationen finden Sie auf der Webseite des Meetings:

              http://pyddf.de/

Viel Spaß !

Marc-Andre Lemburg, eGenix.com

Plone had its 14th birthday last week. Lovingly cared for, with the weight of experience behind it.

First of all, thank you. A lot of work and love has been put into Plone 5. We have built something truly special.

Plone 5 has an all new design. A toolbar on the left for editors, with basically everything you used to be able to do from the edit bar. It lives outside of the theme, so no more problems with a theme that would mess up your edit experience.

Folder contents, with for example bulk adding of tags or changing the state. Upload multiple files.

New TinyMCE, much better than what we had previously. Works responsively, so also on other devices.

We redesigned the control panel. Add-ons page has a complete revamp, much nicer and easier to understand. Social media settings.

Theming has been changed. A Diazo theme as base. You can copy the base theme in the UI and edit it right there. A really powerful tool, it makes the initial theming experience much nicer.

Our default content story is Dexterity now. You can still use Archetypes if you want, for now.

We have finally got recurring events rights, like adding an event that recurs each Sunday until the end of the year.

Plone 5 is accessible, meeting lots of accessibility requirements. You will hear more about that in the keynote tomorrow. New multilingual framework, in the core, working for both Dexterity and Archetypes. So: all the content, all the time, for all the people.

There is lots of room for extending Plone and integrating it into all kinds of other systems.

We have sturdy security, especially with automatic CSRF protection.

Evolution of hackability. In earlier Plones you could work through the web, but this was not considered best practice, because you could not really manage your code nicely. Eric Brehault said: we should build an unhackable core, but add something hackable on top of it. Empower people in all the right places. Customization is a feature. We can use through the web prototyping to rapidly come up with a solution that you can then export.

Annette Lewis is a designer of website on Penn State University. New Plone user. She runs about 40 sites. Base theme and policy products, and the rest is done entirely through the web, with dexterity, diazo, theme editor. "It feels like you can do a lot from the front without touching the web." She now has a two week release cycle for sites.

Mosaic is the successor to the oft-repeated Deco project. Not in core. It allows you to stick content on a page, move it around however you want. Look for plone.app.mosaic. Asko Soukka has been working on this two years straight. In beta. We are close to having it for real. Dexterity plus Diazo plus Mosaic is really powerful.

Look at these Python packages: pep8, pyflakes, flake8, bobtemplates.plone, plone.recipe.codeanalysis. Sysadmins like to be lazy, or should be lazy: he does not want to get a call at midnight to get back to work putting out a fire, because everything is running smoothly, and anyone can do it because everything is setup correctly.

From one colleague I used to need to check all his work, even after explaining, explaining, explaining. There might be tests, maybe not, maybe failing, non standard code, etcetera.

Instead of spending time on real reviewing, it turned into a fight about how to do things, which style to use. So I stopped complaining. I did it differently: let the machines work for you. Let continuous integration tell you all the things.

Are tests passing? Pull requests on Plone are passed through our Jenkins test server.

Does code adhere to a style guide? Is it automatically checked, also by Jenkins or Travis?

QA is about asking why? Why did you make the changes this way? Why did you create this new buildout part? Step back from the line-to-line review and get the full overview. Maybe the change should be in a new package, or the change should not be done at all.

Side note: NASA's take on software development. See http://www.fastcompany.com/28121/they-write-right-stuff

End user or developer documentation. Maybe you create a new option for a buildout recipe, but nobody knows about it unless you document it. Maybe you write everything in English and not everyone can read it. Does it run in all browsers, did you test that? Maybe there is a broken layout, because you forgot to add an upgrade step. Is a dependency missing?

Side note: for checking layouts, see huxley, screenster, PhantomCSS.

So how do we fix it?

Have a style guide and conventions, some documentation on how your team wants the code to look like, what best practices you should adhere to. The best style guide is a coded guide, an automated check, especially as a commit hook. Use the plone.recipe.codeanalysis[recommended] recipe. Use flake8, not only in this recipe, but look at it yourself, integrate it in your editor, look at all the plugins that have been created for it. Maybe create one of your own, it is not hard.

Use a CI (continuous integration) system, it saves time. It is a newcomer's best friend, or even the seasoned veteran's best friend.

Fill your toolbox with zest.releaser, z3c.dependencychecker, lots more. Look for tools, share tools and experiences that help you adhere to the style guides and best practices. Improve and master the tools you use.

Side note: isort. Tool that automatically sorts your imports: https://pypi.python.org/pypi/isort

Some optional flake8 plugins: flake8-copyright, flake8-exact-pin, flake8-naming, flake8-docstrings, flake8-diff, flake8-tuples.

How about manual Quality Assurance? Checking a responsive design is probably something you need to do by hand, or spend time learning some new tools.

Single, one-time cleanups? Fine. But then also add tools to check that the package stays cleaned up, otherwise you keep spending your time on this issue.

Should a pep8 error end with a CI failure? If you have decided that you want to adhere to pep8: yes!

Before QA, you should do some pre-QA. Use bobtemplates.plone to create new Plone packages.

What kind of things can you check? Tests, coverage, plone.recipe.codeanalysis, state of translations, documentation syntax on PyPI, working documentation references and typos, a proper MANIFEST.in to avoid 'brown bag' releases (missing files), undeclared dependencies, accessibility, check javascript on major browsers and screen sizes, commit messages, no XXX or TODO markers in the code, have a change log entry for each fix or feature.

When do you run them? You can do it manually. Have a pre-commit hook. Before a release, but don't throw this job at the one doing the release, but do it yourself as initial committer. A per distribution CI job.

Some ideas. Keep the noise level low and targeted: if the whole team needs to look at a ten megabyte CI report, that is not nice. When you work on pull request branches, you can use gitcommit--amend and gitrebase to restrict the change to a few or even just one single commit. Integrate checks in your editor. Maybe goal for Plone 5.1 to have zero code analysis failures, see http://jenkins.plone.org/job/code-analysis. Check the dependencies of packages. Zuul as project gating system: http://docs.openstack.org/infra/zuul/. Moving to Python3, will that ever be possible without best practices? Can a newcomer release Plone 6, or does it take one person who knows all the small details?

Final thoughts:

• No bike shedding, just use linters.
• No manual steps, let CI warn.
• Always room for improvement, fill your toolbox.
• Always have an eye for quality on every action.

Question: is setting up Jenkins difficult? You can look at https://github.com/plone/jenkins.plone.org which is a buildout, with nothing really specific for core Plone, you can use it for yourself. When I started using it at Der Freitag, it was basically half a day of work to setup.

How to enforce the style guide for every single package? Setup a Jenkins job or step to run this check. See the Github repo above.

Helpful hints:

• Use headings.
• Do not use "Read more" or "Click here". It will just get said over and over again by screen readers. Append the title to "read more" and throw it off the canvas with css so it is not in the way.
• Use big targets for touch devices.
• Use closed captioning on videos.
• Do not flash people, watch out for people with epilepsy.
• No tables, unless it is really tabular data.
• In the viewport tag, do not have a maximum-scale: let people zoom in so they can read it.

Forms:

• Every single field needs a label. I mean it! Just use a for and an id that match. If you click on the label and the field is selected, then you have got it right. You can move labels out of the way with CSS if you want, they will still be read by screen readers.
• Put special instructions in the label, like how you want the date formatted. Put helptext inside the label. Put errors in there too.
• What about placeholders? They do not work consistently in screen readers. You enter the input field and the placeholder is gone, so it never gets read.
• Do not say "required," but "name is required," much clearer.
• Use form validation alerts and go to the input that is invalid.
• Forms need a fieldset and label.
• Use type="tel" when you need to input a telephone number. Use those new types.

Modal windows:

• Check that Escape can close it.
• Have more than just X as label for the close button.
• Get back to where you were on the page after closing.
• Make sure users can tab around only on the modal.

Testing: browser tools.

• AXE: http://www.deque.com/products/axe/ is at least partically open sourced recently.
• Firefox has the Fireeyes plugin. Integrates with Firebug. Does a pretty good job of warning you about accessibility problems.
• Bookmarklet HTML_codesniffer.
• Colour Contrast Analyser for picking color and background-color.
• Color Blindness Simulator.

Testing: screen readers. Use the best combinations.

• JAWS, costs about a thousand dollars. Use with IE
• NVDA for Firefox
• Voice Over, Mac only, use for Safari.
• Talkback, on Android.

Checklist: http://pauljadam.com/wcag20checklist.html

WAI-ARIA: web accessibility initiative, accessible rich internet interfaces. HTML 5 makes a good bit of this redundant.

You really, really need to test. Make some friends that have these issues.

Plone 5: the editing interface needs some love. Everything else looks great.

Can you get automated reporting? At least the contrast checker can be run in Firefox in CI, in robotframework. We have permission from the author to do that.

Why another migration tool for Plone? plone.app.contenttypes already has migration for Archetypes to Dexterity. This atct_migrator is in most cases the right tool. Watch and listen to Philip Bauer's talk (tomorrow) about this: How to upgrade sites to Plone 5.

We have Plone 3.3.5, more than 20,000 items. Custom content types, with archetypes.schemaextender. Merging of content from different sites, cluster of five sites sharing a mounted zeo. This live implementation was actually uninstallable anywhere else, so we could not get a good test migration using the default ways.

interaktiv.migrator gets source as json and pushes it to the target.

Pro:

• import content directly to a newer Plone version.
• migrate our own content types without mapping fields that were already matching
• archetypes.schemaextender data worked fine with this
• This could be done from a live system.
• You get a new and clean database.

Contra:

• Only content is migrated, so settings and users are lost. So you should do something else to get this part of your setup back.
• References have to be migrated at a second run.

I wrote a transmogrifier based workflow to export and import the data.

In the old site there is a configuration form for the migrator, with a target url and for security reasons an api key that must match between source and target. On the target site, you set mappings for types, fields, views.

Live demo, migrating some content, with portlets and local roles on the Sharing tab.

After migration some things still need to be done:

• Go to portal_workflow and update the security settings (button).
• Go to portal_catalog and clear and rebuild the catalog (button).

You can write code to export only some portal_types, or only published items, or only content from one folder.

• interaktiv.migrator is in beta.
• Target audience: developers.
• Does not work yet for collections.
• Only migrates from Archetypes to Dexterity.
• Content gets a different uid currently.
• Needs better documentation, testing, refactoring.

Code:

• https://github.com/jmevissen/interaktiv.migrator
• https://github.com/jmevissen/interaktiv.migrator_p3

Questions:

• Are dates migrated correctly, so creation and modification date? No, but for example we added this for the Creators field and this took only half an hour to implement.
• Which Plone versions? Tested on migrating from Plone 3 or 4 to 5.
• plone.app.contenttypes migrates schemaextender, dates, uids, etcetera. Did you report any issues there? No.

At the start of October, I began a new adventure. I’m the PyCharm and WebStorm Developer Advocate at JetBrains. More of the former, some of the latter. This morning JetBrains published the introductory interview with more detail.

I’m super excited about this. As I said in the article, I’ve been wanting for a while to work with others in a product company, learning about all the ways stuff really gets made. And lucky me, the product I use the most, PyCharm, was looking for someone to do the kind of job I enjoy the most.

For now I’ll still be doing a portion of time with Agendaless on existing work. And of course, I’ll still be in the communities I’m currently in, probably doing more, as JetBrains has legit support for people spending time on open source work. It’s deeply, deeply exciting to think of all the relationships in Python I’ll be working on in the coming years. Lots of good going on that we can support.

Plone has a good back end. On the front end lots of stuff is happening with javascript framework, in the brave new Javascript world. The three of us have some personal opinions on the front.

JSON is used to let front and back end talk to each other, for the client and server architecture.

A json rpc api would do a POST to somehost/api, with a method like create_content and give it a type, title, container UID. This is brittle: as soon as you change a detail in this api, you break this. And you do not notice, until someone uses it, unlike a python error that you often discover when the server fails to start up.

You can use multiple URLs. POST to somehost/api/my-folder. Slightly better, you do not need to pass the UID.

You could use all HTTP methods: GET, POST, PUT, DELETE, using the same url with extra parameters. This is what people talk about when doing web apis: REST. An other way to talk about it, is CRUD: Create, Read, Update, Destroy

New package: plone.rest. Register an http method in zcml for an interface of a content item. In your request, tell Plone that you accept json, and plone.rest will return json to you.

You have REST, which is a set of restraints to what such an api should do, and you have hypermedia. This decouples front and back end, in a way that allows you to change the api and without needing to change your client. The client goes to a single entry point and follows links from there. For a proof of concept we used JSON LD and Hydra. You get json with a list of links to members of a folder, including operations that you can do there.

So what is the current status? We have plone.rest and plone.restapi packages.

Ramon: Plone has lots of layers. The rest api is a way to reduce this. Something like PAS (PluggableAuthService) is in essence simple: get the user, get his global and local roles, check which permissions those roles have, and you are done. So why is it so complex? Especially when you use LDAP, making it slow. We are working on getting and storing this info outside of Plone. Then you can use elasticsearch to get roles and permissions. Rely on LDAP and for example docker to get this info. It is an idea. But for some kinds of installations, this may be a good idea, so if you are looking at this, talk to us.

Nathan: page rendering. User accesses a url, python goes to work, and gives an html response. Zope has a Publisher, which in the end gets you this html, using page templates, Chameleon, portlets, etc, some Diazo and Mosaic included, javascript doing some more transformation at the end. Most frameworks like Angular have their own template engines. With Diazo you could actually do the transformation in the browser if you want, there is code for that. We might no longer care about how forms are rendered, letting this be done by a front end framework. For Mosaic, things stay the same, except the rendering that is moved to javascript. So you move a lot of processing over to the browser. Plone server runs REST api, static css and javascript served by apache or nginx, javascript doing the real rendering. Problems: page loads would not include any html, so that is bad for SEO. But you could use NodeJS for this. And search engines are playing nicer with javascript.

Further topics. Forms done by some javascript framework. Search: not ZCatalog but solr or elasticsearch. Diazo: will we still need that? Dexterity will most likely stay: we will still need content types. Hackability, keep that in mind.

This is work in progress. We are discussing ideas. We are not saying that Plone 6 will look like this. These packages will work with your existing Plone sites and not break anything, so you can test this with whatever javascript framework you like.

Javascript frameworks are a moving target: yes, that is hard. For some projects you may want to use React, for others Angular 1.3 or 2. We will provide the restful api as common ground. People can build on top of that. We are not sure this is where the web is going, but we are pretty sure this is a good way that will work.

We have a good front end that is great for a lot of use cases. We are not breaking that. But we are adding possibilities.

Martin: Web sites are generally becoming more complex, with more features, plugin possibilities. If others can talk to you this helps. What matters is accessibility and integrateability, making it nicer for others. If we design the api really good, it can make Plone really successful.

This blog posts shows how to build a Twitter bot using Google Spreadsheets as data source in Python.

The service presented here was originally created for a friend of mine who works in Megacorp Inc. They have a marketing intelligence department that is filling out stalked information about potential customers. This information stored in Google Spreadsheet. Every day a new spreadsheet arrives to a folder. Then my friend proceeds to go through all of the leads in the spreadsheet, check who have a Twitter account and harass them in Twitter about Megacorp Inc. products.

To make my friend jobless I decided to replace his tedious workflow with a Python script. Python is a programming language for making simple tasks simple, eliminating the feeling of repeating yourself with as little lines as possible. So it is a good weapon of choice for crushing middle class labor force participation.

The bot sends two tweets to every Twitter user. Timing between tweets and the second tweet is randomized, just to make sure that no one could not figure out in a blink of an eye that they are actually communicating with a bot.

The ingredients of this Twitter bot are

• Python 3.4+– a snake programming language loved by everyone
• gspread– a Python client for Google Spreadsheets making reading and manipulating data less painful
• tweepy– A Twitter client library for Python
• ZODB– An ACID compliant transaction database for native Python objects

The script is pretty much self-contained, around 200 lines of Python code and 3 hours of work.

1. Authenticating for third party services

The bot uses OAuth protocol to authenticate itself against Google services (Google Drive, Google Spreadsheet) and Twitter. In OAuth, you arrive to a service provider web site through your normal web browser. If you are not yet logged in the service asks you log in. Then you get this page where it asks authorize the app. Twitter authentication is done in a separate script run tweepyauth.py which asks you enter the pin number shown on Twitter website. Google API client does things different and spins up a local web server running in a localhost port. When you authorize on Google services it redirects you back to the local webserver and the script grabs the authentication token from there.

The script stores authentication tokens in JSON files You can run the script on your local computer first to generate JSON files and then move it to the server where a web browser for authentication is not possibly available.

2. Maintaining persistent state

The bot needs to maintain a state. It needs to process a new spreadsheet every day. But on some days the bot might not be running. Thus, it needs to remember already processed spreadsheets. Sometimes the spreadsheets may contain duplicate entries of the same Twitter handle and we don’t want to harass this Twitter user over and over again. Some data cleaning is applied to the column contents, as it might be raw Twitter handle, HTTP or HTTPS URL to a Twitter user – those marketing intelligence people are not very strict on what they spill in to their spreadsheets.

The state is maintained using a ZODB. ZODB is a transaction database, very robust. It is mature, probably older than some of the blog post readers, having multigigabyte deployments running factories around the world. It can run in-process like SQLite and doesn’t need other software running on the machine. It doesn’t need any ORM as it uses native Python objects. Thus, to make your application persistent you just stick your Python objects to ZODB root. Everything inside a transaction context manager is written to the disk or nothing is written to the disk.

As a side note using Google Spreadsheets over their REST API is painfully slow. If you need to process larger amounts of data it might be more efficient to download the data locally as CSV export and do it from there.

3. Usage instructions

This code is exemplary. You can’t use it as you do not have correct data or access to data. Use it to inspire your imagination. However if you were to use it would happen like this:

• Follow the official Python package installation guide to install the dependencies
• Register Twitter app. Leave callback URL empty.
• Register a Google services App.
• Run tweepyauth.py to get Twitter tokens stored in twitter_oauth.json.
• Run bot once on your local computer and authenticate it against Google services and write client_secrets.json.
• See that the bot starts working.
• Move the whole stuff to a server.
• Leave it running in a loop on Bash prompt forever: $ while true; do python chirper.py; sleep

4. Source code

chirper.py

"""

Installation:

    pip install --upgrade oauth2client gspread google-api-python-client ZODB zodbpickle tweepy iso8601
"""

import time
import datetime
import json
import httplib2
import os
import sys

# Authorize server-to-server interactions from Google Compute Engine.
from apiclient import discovery
import oauth2client
from oauth2client import client
from oauth2client import tools

# ZODB
import ZODB
import ZODB.FileStorage
import BTrees.OOBTree
from persistent.mapping import PersistentMapping
import random
import transaction

# Date parsing
import iso8601

# https://github.com/burnash/gspread
import gspread

# Twitter client
import tweepy

try:
    import argparse
    flags = argparse.ArgumentParser(parents=[tools.argparser]).parse_args()
except ImportError:
    flags = None


# We need permissions to drive list files, drive read files, spreadsheet manipulation
SCOPES = ['https://www.googleapis.com/auth/devstorage.read_write', 'https://www.googleapis.com/auth/drive.metadata.readonly', 'https://spreadsheets.google.com/feeds']
CLIENT_SECRET_FILE = 'client_secrets.json'
APPLICATION_NAME = 'MEGACORP SPREADSHEET SCRAPER BOT'
OAUTH_DATABASE = "oauth_authorization.json"

FIRST_TWEET_CHOICES = [
    "WE AT MEGACORP THINK YOU MIGHT LIKE US - http://megacorp.example.com",
]

SECOND_TWEET_CHOICES = [
    "AS WELL, WE ARE PROBABLY CHEAPER THAN COMPETITORCORP INC. http://megacorp.example.com/prices",
    "AS WELL, OUR FEATURE SET IS LONGER THAN MISSISSIPPI http://megacorp.example.com/features",
    "AS WELL, OUR CEO IS VERY HANDSOME http://megacorp.example.com/team",

]

# Make sure our text is edited correctly
for tweet in FIRST_TWEET_CHOICES + SECOND_TWEET_CHOICES:
    assert len(tweet) < 140

# How many tweets can be send in one run... limit for testing / debugging
MAX_TWEET_COUNT = 10


# https://developers.google.com/drive/web/quickstart/python
def get_google_credentials():
    """Gets valid user credentials from storage.

    If nothing has been stored, or if the stored credentials are invalid,
    the OAuth2 flow is completed to obtain the new credentials.

    Returns:
        Credentials, the obtained credential.
    """    credential_path = os.path.join(os.getcwd(), OAUTH_DATABASE)

    store = oauth2client.file.Storage(credential_path)
    credentials = store.get()
    if not credentials or credentials.invalid:
        flow = client.flow_from_clientsecrets(CLIENT_SECRET_FILE, SCOPES)
        flow.user_agent = APPLICATION_NAME
        if flags:
            credentials = tools.run_flow(flow, store, flags)
        else: # Needed only for compatability with Python 2.6
            credentials = tools.run(flow, store)
        print('Storing credentials to ' + credential_path)
    return credentials


def get_tweepy():
    """Create a Tweepy client instance."""    creds = json.load(open("twitter_oauth.json", "rt"))

    auth = tweepy.OAuthHandler(creds["consumer_key"], creds["consumer_secret"])
    auth.set_access_token(creds["access_token"], creds["access_token_secret"])
    api = tweepy.API(auth)
    return api


def get_database():
    """Get or create a ZODB database where we store information about processed spreadsheets and sent tweets."""    storage = ZODB.FileStorage.FileStorage('chirper.data.fs')
    db = ZODB.DB(storage)
    connection = db.open()
    root = connection.root

    # Initialize root data structure if not present yet
    with transaction.manager:
        if not hasattr(root, "files"):
            root.files = BTrees.OOBTree.BTree()
        if not hasattr(root, "twitter_handles"):
            # Format of {added: datetime, imported: datetime, sheet: str, first_tweet_at: datetime, second_tweet_at: datetime}
            root.twitter_handles = BTrees.OOBTree.BTree()


    return root


def extract_twitter_handles(spread, sheet_id, column_id="L"):
    """Process one spreadsheet and return Twitter handles in it."""    twitter_url_prefix = ["https://twitter.com/", "http://twitter.com/"]

    worksheet = spread.open_by_key(sheet_id).sheet1

    col_index = ord(column_id) - ord("A") + 1

    # Painfully slow, 2600 records = 3+ min.
    start = time.time()
    print("Fetching data from sheet {}".format(sheet_id))
    twitter_urls =  worksheet.col_values(col_index)
    print("Fetched everything in {} seconds".format(time.time() - start))

    valid_handles = []

    # Cell contents are URLs (possibly) pointing to a Twitter
    # Extract the Twitter handle from these urls if they exist
    for cell_content in twitter_urls:

        if not cell_content:
            continue

        # Twitter handle as it
        if "://" not in cell_content:
            valid_handles.append(cell_content.strip())
            continue

        # One cell can contain multiple URLs, comma separated
        urls = [url.strip() for url in cell_content.split(",")]

        for url in urls:
            for prefix in twitter_url_prefix:
                if url.startswith(prefix):
                    handle = url[len(prefix):]

                    # Clean old style fragment URLs e.g #!/foobar
                    if handle.startswith("#!/"):
                        handle = handle[len("#!/"):]

                    valid_handles.append(handle)

    return valid_handles


def watch_files(http, title_match=None, folder_id=None) -> list:
    """Check all Google Drive files which match certain file pattern.

    Drive API:

    https://developers.google.com/drive/web/search-parameters

    :return: Iterable GDrive file list
    """    service = discovery.build('drive', 'v2', http=http)

    if folder_id:
        results = service.files().list(q="'{}' in parents".format(folder_id)).execute()
    elif title_match:
        results = service.files().list(q="title contains '{}'".format(title_match)).execute()
    else:
        raise RuntimeError("Unknown criteria")

    return results["items"]


def scan_for_new_spreadsheets(http, db):
    """Check Google Drive for new spreadsheets.

        1. Use Google Drive API to list all files matching our spreadsheet criteria
        2. If the file is not seen before add it to our list of files to process
    """    # First discover new spreadsheets

    discovered = False

    for file in watch_files(http, folder_id="0BytechWnbrJVTlNqbGpWZllaYW8"):
        title = file["title"]
        last_char = title[-1]

        # It's .csv, photos, etc. misc files
        if not last_char.isdigit():
            continue

        with transaction.manager:
            file_id = file["id"]
            if file_id not in db.files:
                print("Discovered file {}: {}".format(file["title"], file_id))
                db.files[file_id] = PersistentMapping(file)
                discovered = True

    if not discovered:
        print("No new spreadsheets available")


def extract_twitter_handles_from_spreadsheets(spread, db):
    """Extract new Twitter handles from spreadsheets.

        1. Go through all spreadsheets we know.
        2. If the spreadsheet is not marked as processed extract Twitter handles out of it
        3. If any of the Twitter handles is unseen before add it to the database with empty record

    """    # Then extract Twitter handles from the files we know about
    for file_id, file_data in db.files.items():

        spreadsheet_creation_date = iso8601.parse_date(file_data["createdDate"])

        print("Processing {} created at {}".format(file_data["title"], spreadsheet_creation_date))

        # Check the processing flag on the file
        if not file_data.get("processed"):
            handles = extract_twitter_handles(spread, file_id)

            # Using this transaction lock we write all the handles to the database once or none of them
            with transaction.manager:
                for handle in handles:
                    # If we have not seen this
                    if handle not in db.twitter_handles:
                        print("Importing Twitter handle {}".format(handle))
                        db.twitter_handles[handle] = PersistentMapping({"added": spreadsheet_creation_date, "imported": datetime.datetime.utcnow(), "sheet": file_id})

                file_data["processed"] = True


def send_tweet(twitter, msg):
    """Send a Tweet.
    """    try:
        twitter.update_status(status=msg)
    except tweepy.error.TweepError as e:
        try:
            # {"errors":[{"code":187,"message":"Status is a duplicate."}]}
            resp = json.loads(e.response.text)
            if resp.get("errors"):
                if resp["errors"][0]["code"] == 187:
                    print("Was duplicate {}".format(msg))
                    time.sleep(10 + random.randint(0, 10))
                    return
        except:
            pass

        raise RuntimeError("Twitter doesn't like us: {}".format(e.response.text or str(e))) from e

    # Throttle down the bot
    time.sleep(30 + random.randint(0, 90))


def tweet_everything(twitter, db):
    """Run through all users and check if we need to Tweet to them. """    tweet_count = 0

    for handle_id, handle_data in db.twitter_handles.items():

        with transaction.manager:

            # Check if we had not sent the first Tweet yet and send it
            if not handle_data.get("first_tweet_at"):

                tweet = "@{} {}".format(handle_id, random.choice(FIRST_TWEET_CHOICES))

                print("Tweeting {} at {}".format(tweet, datetime.datetime.utcnow()))
                send_tweet(twitter, tweet)
                handle_data["first_tweet_at"] = datetime.datetime.utcnow()
                tweet_count += 1

            # Check if we had not sent the first Tweet yet and send it
            elif not handle_data.get("second_tweet_at"):

                tweet = "@{} {}".format(handle_id, random.choice(SECOND_TWEET_CHOICES))

                print("Tweeting {} at {}".format(tweet, datetime.datetime.utcnow()))
                send_tweet(twitter, tweet)
                handle_data["second_tweet_at"] = datetime.datetime.utcnow()
                tweet_count += 1

        if tweet_count >= MAX_TWEET_COUNT:
            # Testing limiter - don't spam too much if our test run is out of control
            break


def main():

    script_name = sys.argv[1] if sys.argv[0] == "python" else sys.argv[0]
    print("Starting {} at {} UTC".format(script_name, datetime.datetime.utcnow()))

    # open database
    db = get_database()

    # get OAuth permissions from Google for Drive client and Spreadsheet client
    credentials = get_google_credentials()
    http = credentials.authorize(httplib2.Http())
    spread = gspread.authorize(credentials)
    twitter = get_tweepy()

    # Do action
    scan_for_new_spreadsheets(http, db)
    extract_twitter_handles_from_spreadsheets(spread, db)
    tweet_everything(twitter, db)


main()

tweepyauth.py

import json
import webbrowser

import tweepy

"""
    Query the user for their consumer key/secret
    then attempt to fetch a valid access token.
"""

if __name__ == "__main__":

    consumer_key = input('Consumer key: ').strip()
    consumer_secret = input('Consumer secret: ').strip()
    auth = tweepy.OAuthHandler(consumer_key, consumer_secret)

    # Open authorization URL in browser
    webbrowser.open(auth.get_authorization_url())

    # Ask user for verifier pin
    pin = input('Verification pin number from twitter.com: ').strip()

    # Get access token
    access_token, access_token_secret = auth.get_access_token(verifier=pin)

    data = dict(consumer_key=consumer_key, consumer_secret=consumer_secret, access_token=access_token, access_token_secret=access_token_secret)
    with open("twitter_oauth.json", "wt") as f:
        json.dump(data, f)

 Subscribe to RSS feedFollow me on TwitterFollow me on FacebookFollow me Google+

Use the PageSpeed tool to measure your page and get hints for improvement.

Move javascript from the head to the bottom.

• You can do that with Diazo rules or through the web.
• But you can get flashes of unstyled content. Check this. There is no one silver bullet solution. See if you can do some transformations not in javascript but in a different way with css.
• Random javascript could break, if it expects jQuery or other javascript already available, especially if they get loaded inline. Usually loading on document ready should solve this.
• Plone core may move the javascript to the bottom some day, there is discussion.

On Plone 4, with portal_css and portal_javascripts, you can reorder the css and javascript to fit more together. Currently you often have for example member.css in the middle, restricted to only members, and this condition splits the top and bottom css files in two. Move the member.css and then anonymous users need just one combined css.

Check the caching rules, especially for files like the favicon and the logo that will almost never change.

Online tools and background information for css:

• Use http://csslint.net to check your css.
• Another tool: http://www.stylestats.org
• Make your css as simply as possible: http://csswizardry.com/2015/04/cyclomatic-complexity-logic-in-css/

Optimize your javascript:

• PhantomJS-based web performance metrics collector and monitoring tool: https://github.com/macbre/phantomas

Document decisions and best practices in a style guide.

• Eric Steele: "I think we should set up a style guide for the core UI." See https://github.com/plone/Products.CMFPlone/issues/589
• There are automated style guides, for example kss-node, StyleDoccu, DocumentCSS, Hologram. Many need different stacks, and the generated guide may come out very ugly.
• The EEA style guide: http://www.eea.europa.eu/code/design-elements
• Plone has the test_rendering page that has all (or most) design elements of Plone. We could run https://github.com/filamentgroup/X-rayHTML over this. Add a data attribute and then this shows the generated page and makes the html itself visible.
• Make it easy to maintain, by having the style guide in the same website that it is meant for.
• Further reading: http://styleguides.io
• Documentation: add comments in css explaining why you do something.

We need to monitor for regressions.

• Automate tests with Travis, using headless browsers.
• psi command line tool
• Grunt-PerfBudget

In closing:

• Measure your site performance regularly.
• Presentation by Addy Osmani: https://speakerdeck.com/addyosmani/css-performance-tooling
• Style guides from github: https://github.com/styleguide

We, Plone developers, are architects. We build a sand castle. Then users play with it, love it, and break it. Children play with it and it gets broken. (Watch short video Les Vacances.) Conflict: users love it and want new features, and we despair. So we build a nuclear power plant. Unbreakable, but the users cannot play with that any longer. Well, maybe terrorists can play with it, but they are not are target audience.

Pharmakon: any medicine is also a poison.

Build an inflatable castle instead: it is bouncy (more fun for the children), robust (more fun for us), and terrorists do not like it.

Hackability is a feature. It is not a flow. It must be provided as a tool, something on top. If we lose this, we will lose part of our audience. We should have a hacking tool, not a hackable system. A moderen and pleasant web interface.

And we have it: the ZMI. Well... Using framesets, really?

We have the Plone theming editor. It is already a hacking tool. Non-Plone experts can change the entire theme. The Diazo theory behind it, is: "We write XSLT so you don't have to." In reality I am writing tons of XSLT, because there is a really detailed theme layout and lots of pieces of content that need to be hand-picked into it.

We need on-the-fly content changes with before and after: on-the-fly content insertion, also from 'remote' to insert a small page. We have this working, but it is not merged yet.

We want more. Create our own chunks of content and scripts when you only know Python and javascript, and some simple diazo to include it in your rules. This is what Rapido offers. Rapido is not a new Plomino, it is something different.

Rapido lives in the theme. Editable in the Plone UI or in the source. And then we use Diazo to inject it into Plone.

It is very simple. No zca, no zpt (or any templating language), no jbot, just html and python.

Demo. [Looks quite nice, check the video once it is uploaded.]

We use souper for persistance. You can store on the context, or on the user, or what you want

Is it secure? Python scripts are executed by zope.security.untrustedpython. All the regular security stack is applied, like current user privileges, CSRF policy, PostOnly. Just as secure as Diazo.

Rapido comes with a complete REST api with GET, POST, PUT, DELETE, PATCH. It is a ready-to-use JSON backend.

Rapido is Mosaic compliant, but does not depend on it. We just make a tile available to insert a Rapido element.

Questions?

Yes, you can import and export it and include it in a package or have it as a zip file, so you have it on the file system.

When will it be included in core? Not now. We may talk with the security team.

Debugging? More difficult: you cannot add a pdb, same as in the Python skin scripts. We can make something possible, at least show on which line something goes wrong.

Philip: You still need to know Plone when you access for example the portal root in the code you create. I am not sold on it, just like I was never a fan of .cpy files. You lose debuggability. But yes, it lowers the barrier of entry. I am not the target audience.

There are of course permissions, so you can decide who can use this. And you can use access lists in a yaml file.

ZODB is pluggable, with lots of options for storage: FS (Data.fs accessed by single zope instance), ZEO, ZRS, demostorage, RelStorage.

RelStorage uses as backend MySQL (gave us problems to setup), Oracle (expensive), PostgreSQL (just right for us).

It can be up to twice as fast, mainly because transactions are faster, so you can have more concurrent writes. The limit is always the transaction conflicts.

PostgreSQL is an active open source project, mature, flexible, fast, standard. Sysadmins can handle it just fine.

You can use RelStorage in history-free storage. With history you can undo transactions in the ZMI. With large sites, history-free makes more sense, because usually so many write are done that mostly anything you want to undone has already been overwritten, so undo will not work.

You can use memcached. You need to poll the database for changes in intervals, with ZEO this happens automatically. You can replicate the database with standard tools. You can migrate from RelStorage to FileStorage and back.

Caching. Different threads and instances can use the same memcached client, saving memory. Use an own memcache per zope client machine: so on that same machine. And do not let this same memcache handle for example LDAP.

You should use blobstorage, so check your code, your content types: you do not want files in either FileStorage or RelStorage. [Note from Maurits: PostgreSQL has large object support, which works, but we switched to a shared filesystem blob storage for a client.]

You can split in edit and read-only clients and use a different poll interval. Use 0.5 to 10 seconds for read-only.

Do have a look at configuring and tuning PostgreSQL.

Now lots of time for questions.

Compared to normal setup, you need to replace the ZEO server by RelStorage, add a memcached server on each machine that has the Zope client. Configure the Zope clients to talk to RelStorage.

We could sprint on it to use more options of Postgres 9, making it even faster.

Original goal maybe ten years ago was that some customers required a specific relational database. Things have changed, NoSQL storage is more common now. But how is it possible that it may still be twice as fast as ZEO, as ZEO is quite simple actually? The portal_catalog is part of it probably. The code is hard, takes time to get into. In ZEO things could be tweaked. But for policy reasons it can still be handy to be able to say to an IT department: just give us a Postgres database.

Transactions are still serialized, not concurrent. Switching from pickled format to JSON may make it easier to use more Postgres features.

Can't we change ZEO so you can have a memcache between the zeo client and zeo server, like for RelStorage? So you need less disk cache and can use the memory more.

Avoid write on read.

Python 3? Does not currenly work as far as I know, but should not be that hard. It is just plain Python. ZODB itself works on Python 3.

This is part two of a talk I did in the Brazil conference two years ago. Recap:

• Every non-trivial upgrade should be approached as a relaunch. Look at content, site structure, design, underlying packages, different languages to use in the site.
• The primary challenge is not development, but communication and project management. It is not just a new version number.
• Bring time. And space. You need to do all steps multiple times. Get lots of disk space. Keep several copies of the database so you do not have to start over at point 1 again.
• Expect everything to break. Templates, python files, add-ons, etc.
• Don't experiment, document. Write every problem and solution down. It is not done, you have just done it on a test migration on your laptop, you will have to do it later on the live site again.
• Write your code as if your own kids will inherit the code one day. You don't want them to hate you. You need to understand what it does one year from now.
• Write upgrade-steps in your add-ons, so you can just run it and not have to do all kinds of stuff manually.
• Divide and conquer. Once one step works, commit the changes and make a copy so you can worry about the next step without worrying about the old steps.
• Don't try more than two new technologies at once.
• Use the help. Use plone.api. Use ftw.upgrades, several nice helper steps.

Now upgrades from Plone 4 to 5.

Why would you want to migrate? Why would the client want it? Do you simply want to play with shiny new tools? If the site is running just fine, they can stay on Plone 4. Are there any new things in Plone 5 that they really need or want?

• Great responsive design out of the box. But they may already have it in Plone 4.
• Newer TinyMCE? The editor just always has some problems, no matter which Plone version.
• Security? Hard to sell.
• Prototyping can be easier, say for dexterity, but how often to you prototype on a live site?
• New folder contents, new UI: yes, many will like this.

So do not force your client.

Demo of upgrade of small site. Includes optional upgrade from Archetypes to Dexterity content types.

You lose one thing: the content history, so the CMFEditions revisions. That is really hard to do.

Demo worked fine. It can't be that easy, right? When the old site has lots of configuration and installed add-ons, it gets trickier. Many add-ons still need changes.

• LinguaPlone is hard to get rid of. You need to migrate this to plone.app.multilingual 1.x in Plone 4 already.
• Theming is tricky. I got an error when trying rendering anything. I saw I had to install plone.app.theming manually.

Use PDBDebugMode to debug tracebacks.

Please work on problems like this, document problems, fix problems, report them.

Read the upgrade guide and work on it during this sprint: http://docs.plone.org/manage/upgrading/version_specific_migration/p4x_to_p5x_upgrade.html We will upload this to a Google doc so more people can work on it this weekend and get it properly integrated later.

Disable your custom theme before migrating, otherwise it will break. Use the new Barceloneta diazo theme as basis and work from there.

Archetypes content types will still work just fine, maybe some less nice widgets in editing. You should migrate to dexterity.

Move everything to the portal registry. All settings are there. So no longer set something in for example the portal_properties.

Before you work on your custome code:

• Figure out if you still need it, maybe the problem it tried to solve is already gone.
• Use plone.api for anything you write new or that you change.
• Formlib is kind-of gone. Custom forms should be done with z3c.form.
• Portlets are also moved to z3c.form.
• Check any templates that you override, take over changes, or remove the override.
• For custom migrations from Archetypes to Dexterity, look at these views: pac_installer, atct_migrator, migrate_from_atct, custom_migration.

I have been involved with ICT accessibility since the mid nineties. I was a developer of the FreeBSD project. I am a Unix server admin, web admin, accessibility advocate, podcaster. And I have used Plone for about five years.

I am totally blind, almost since birth. But I can use a PC and such.

There is no universally accepted definition of accessibility, as far as I know. It depends on the context. For me: information or service can be accessed by 'everyone' with virtually equal costs.

Who is everyone?

• With or without disability.
• Age.
• Situational: if you are driving a car, you cannot pay much attention to visual clues; in a quiet library you should probably not use audio; maybe you temporarily cannot use your hands.
• With different devices, not just PCs.
• Language, not everyone speaks English or Japanese.

So how about the costs of access?

• Can be simply a financial cost.
• Time: without hands, you may need more time for some actions.

You first need to know content is there: I cannot see an image without an alternative text.

Key to accessibility is:

• Make everything machine-readable.
• Help assistive technologies.

What are assistive technologies?

• Software or hardware to assist users with specific needs. For example screen readers. Several operating systems have packages for this installed by default.
• Magnifying part of the screen.
• Alternative input devices. Keyboards with very large keys. Complicated switch interfaces.

Machine readable:

• If you show an image of a character, versus text data, most people will be able to see them both. Machines cannot, unless they have image recognition.
• You can make text visually larger and bolder, or you can explicitly enclose it in an h1 tag.

What needs to be machine readable?

• Content to be presented. Have alternatives for images.
• Roles and states of UI elements. Identify menu items as menu items.
• And everything else if you can.

Besides this:

• Is there enough color contrast?
• Are the characters large enough? I can still 'see' 1-point fonts, but most people cannot.
• Can text be resized by the visitor?
• Make it easier to operate. Timeouts, like you need to finish a form within ten minutes otherwise you are logged out, can be hard for people with a disability on their hands.
• Do not autoplay audio, because then I cannot hear my screen reader anymore, so I cannot even stop it and will just leave the page.

So what specifically can you do? See Rob Porter's presentation: http://maurits.vanrees.org/weblog/archive/2015/10/rob-porter-how-you-can-become-an-accessibility-superhero

From a technical aspect, you are opening up to a broader audience and an increasing variation of access devices.

There may be social or legal aspects. There is an aging population that needs reading glasses. Don't worry about me there. The US has legislation, other countries too.

Why Plone? Possible choices for me were Movable Type, Drupal, Plone. What did I want?

• I should be able to manage it, edit it, and use it.
• Multilingual support.
• I want an inexpensive license.

Movable Type was not a practical choice.

Drupal: I did not want to use PHP. Actually it is doing really well for accessibility.

From a blind programmer's perspective: Python sucks. You have to count the spaces. But still better than PHP's incompatible upgrades and stuff.

Accessibility in Plone:

• Really good up to around Plone 3 days.
• In Plone 4 it was outdated.
• Plone 5: not perfect, but a bright future.

The accessibility info page was well written at first, but was forgotten after a while.

Up to Plone 4.x: Kupu was inaccessible, at least for me. No critical issues with generated content. With third-party products it was a different story.

Plone 5:

• Many outdated techniques have been removed.
• Accessibility page rewritten.
• Recent TinyMCE is completely accessible. Really good. I don't have to disable it anymore.
• I see increasing interest and attention from developers.

Room for improvement:

• Javascript widgets need to integrate WAI-ARIA: use roles and states for assistive technology.
• Some issues with color contrast of default theme.
• Diazo editor is not accessible.
• Need to raise awareness in community.
• Well written documentation for theme editing is needed.

Then Plone 5 could be a good choice.

If you are using assistive technology, give more feedback to the developers please.

Look at the web content accessibility guidelines: http://www.w3.org/TR/WCAG20/

Slides: https://www.accessibility.org/~max/nap/talks/201510-plone.html

Twitter: @mnakane (English), @ma10 (Japanese)

I came to know Plone during my Google Summer of Code this year. I never used Plone before, but I started loving it.

My project was about transforming safe html, to do that with lxml now.

Plone is one of the most frequent participants in the Google Summer of Code, which I found a plus. The people on IRC were really helpful during the initial phase. You really helped me a lot! And there was awesome documentation. If you start from the first page, and on and on, then it will be a nightmare, but if you know what you are looking for, it is awesome.

I came across this ticket: https://dev.plone.org/ticket/14929. Proposed by Tom Gross, who had earlier tried it, but ran into many test failures. plone.transform was an earlier try as well, during GSOC'07. We had discussions, about moving it from how CMFDefault did it, to using lxml. Thought about the intelligenttext transform too.

I had difficulties working with Plone. It is not easy for anyone who is new. The code base is vast. The first two months it felt very alien. Difficult for me to figure out where to start. The safe_html of portal_transforms is quite old, and I was not used to those libraries, but I did need to know what this code was actually doing.

So lots of difficulties, but also lots of fun. I learnt how to work in a team, often asking 'can you help me?' I learnt new things about Plone, and how fun it is. Learnt how to write efficient code, and how to document how I did it. Pep8, indentation, etcetera, producing good code. Test driven development, I never tried that before. This is one of the best parts of Plone. I got to interact with cool people. They taught me a lot, including some new beers of Europe.

Main goal: get rid of the transform we were using from CMFDefault.

The new filter. Lots of things required to setup the add-on, generic setup, browser view, control panel, interface, an uninstall profile. We have automatic registration and unregistration, done when installing the add-on. Package is: experimental.safe_html_transform

Code: https://github.com/collective/experimental.safe_html_transform

New releases of Plone were done, and suddenly tests started failing. plone.app.widgets was not pinned and got pulled in, and the latest version was trying to pull in the latest CMFPlone version, which was too high at that point, giving an version conflict during buildout.

We created a new control panel, with separate permission.

The new transform uses lxml. It converts the document into the tree form and parses each node of the tree. It checks if the tags or attributes are safe, strips or removes them if not. lxml is faster than the previous solution from CMFDefault.

We used unit tests to test the transform, with sample html. Also automatic robot tests for the control panel.

Now TinyMCE should use our new transform. It uses getToolByName to get the portal_transforms to get the registered transform by its name: it wants safe_html. So we renamed the transform to safe_html.

One thing left: integration of control panel and script.

I learned test driven development, collaboration with people on a project, also a big project, how to understand errors from the error logs, how to keep logs for huge projects (blogging, task managers, etc), because you will forget what you have done five months ago.

Plans: I started loving Plone, so I look forward to learning more about it. First I will finish this project.

Most of the people in India do not use Plone. In my university no one used it. That is a barrier for picking up such a project from GSOC. More awareness is needed, so people are not stuck with Wordpress and need to learn the basics of Plone still.

I never took a training of Plone, that would have helped, like I did this week.

I will be here for one day of the sprints. I would like to work with you.

Barceloneta is a neighbourhood in Barcelona. It is also the default diazo theme in Plone 5.

Every resource has its own purpose, like scaffolding, normalizing. We have variables.plone.less that defines all the variables that we use in the theme. You can change them in the Plone UI, or on the file system.

It is based on bootstrap 3, but is not dependent on it. We took a 'photo' of bootstrap at some point, and went from there. We prepended plone- whenever possible. Oriol Marti and Albert Casado Celma helped a lot creating the Barceloneta theme. The original bootstrap resources are included on Plone.

We preserved Plone legacy classes and ids, to try not to break old themes. Things like: .portlet, .formControls are still there.

We updated breadcrumbs, images, portlets, alerts, control panels, buttons, almost everything.

New viewlet manager: plone.mainnavigation, with the folder sections in it.

We used http://fontello.com to build a customized set of icons. Used in plone bundle and Barceloneta. See Products.CMFPlone/static/fonts. Just load config.json in Fontello and start adding your own icons.

Plone Toolbar. See plone.app.layout and plone.app.contentmenu. Small changes since last year. You can change the orientation in the site control panel, so make it horizontal at the top. Do not forget to make space for the toolbar in your Diazo theme.

For the control panel we also used a Fontello font. Add a CSS class with the normalized name of your new control panel item, something like:

icon-controlpanel-FilterSettings: before {content: \e844}

Easy backend customization, by reusing Barceloneta resources. You can use the backend.xml Diazo rules and related css. In the future you can use the Plone Intranet theme switcher.

We use the new Resource Registries introduced in Plone 5. A resource is a collection of less files and javascript. So it is more like a web component for Plone.

Then you can register a bundle: a collection of resources.

You can do easy customization of ++plone++ resources. Why only allow exactly those resources and not others? I think it can be very useful.

Automatic variables:

@import url("@{mockup-patterns-select2}");

This is available for every resource, used in compilation.

You should no longer fill the javascript_head_slot or style_slot in a template, but in a view do add_resource_on_request, or add_bundle_on_request. Use this if you are going to use it just in one view.

Console script helpers:

bin/plone-compile-resources
bin/plone-generate-gruntfile

Call them with:

--site-id=myplonesite --bundle=mybundle

A pure Diazo theme is easy for a designer to jump in. Distributed through a zip file for import and export. Diazo bundles preserves this pureness. This uses new attributes for the manifest.cfg file of the theme, like enabled-bundles, tinymce-content-css, production-js.

Some best practices:

• Probably no best practices that fit every scenario.

• Reuse Barceloneta as far as you want. There is an optional Barceloneta profile to register its resources. If you want this, then in your metadata.xml add this as dependency:

  profile-plonetheme.barceloneta:registerless
  

• You are not tied to bootstrap, you can use what you want. Bootstrap is probably easier though.

• Don't be obsessed with the number of requests your site is doing: HTTP 2.0 is almost here, which helps a lot. It is still important, but it will be less so.

• Skin layers are not allowed. Okay, they are still there, but please don't use them. Use plone.resource instead.

• For overriding resources: use z3c.jbot.

• You can use the extend feature:

  .context:extend( .btn-primary) { }
  

• Adapt the grid in Diazo if you want, or via a custom transform like Mosaic is doing.

• Do not create a theme from scratch yourself, but use bobtemplates.plone. Understand the structure of the default theme.

• TinyMCE: custom generated style sheets. Define the new Diazo Tiny content specific css in manifest.cfg. TinyMCE content templates are now in core.

• You can define an nginx passthrough for static plone.resources, so ones that are ++something++. Point it to the local file system for that package, probably using collective.recipe.omelette.

Useful tools:

• Chrome reloader: Eric Brehault's button.
• ?diazo.off=1 and ?diazo.debug=1 in the url
• @@test_rendering page with the common rendering items of Plone.

TTW (Through The Web) is back. You can change the logo in the UI now. plone.app.theming has more power than ever before. It can attract new people. It is easy to do things that should be easy, so non technical people can try out Plone fast. And complex things are possible.

You can try out collective.jbot as customization story, not to be confused with z3c.jbot. Should work on Plone 5 soon, please help.

Overriding ++namespace++ would be nice, especially for sites where you have different parts that want to look slightly differently.

If five add-ons add five bundles, this may be much. We may need to find a better way for this. Also, if all those bundles depend op jQuery, you will get five big bundles, plus the default big plone one.

I am working with Plone since 2006. Also involved in writethedocs.

I like to complain. Sometimes too picky? Working with Plone documentation I can get grumpy, sad, annoyed, angry. I will give a heavily opinionated talk. Please do not feel offended if I talk negatively about the documentation (or lack of it) of your package.

This talk is from a user point of view. You must realize that a good product is not just about the code. You look at the whole package. If the docs are broken, your product is broken.

On the Plone 5 release day, we had links to at least three different versions of Plone in the installer part. Confusing. Lots of old and not working or not checked code examples. Lots of confusing mixups between Archetypes and Dexterity. There were pending (not published) docs about theming. If you are a Plone consultant, you can find your way, or know who to ask, but as a user... No note that lots of add-ons are not working with Plone 5 yet. No note telling that some of the docs are not updated yet. The answer to a lot of questions on irc was: "The docs about X are not finished yet."

We forgot all of that, or had no time: our fault.

The technical stack is great. Plone 5 is ready as consultant-ware. For normal humans, the docs were not ready yet. Strictly spoken, we forgot to care about normal users.

You are done when it works and it is elegantly documented. Do at least something.

Documentation is marketing. It gets you new users much quicker. It can save a lot of time, effort, money.

Think about the process of documentation for your project as the true act of creation. Documentation Driven Development. I did not make this up!

1. You write Documentation.
2. You write Tests.
3. You write Code.

Simple!

You give yourself the chance to think the project through, without getting bogged down in a detail that you will rip out five minutes later.

Developer documentation is not end-user documentation.

It is much easier to write documentation at the beginning when you are still happy and excited about the project, instead of at the end when you are glad you have finally squashed most bugs and can ship it.

It is much easier to have a discussion based on something that you have written down.

Create an easy and friendly landing page. Make it easy to report a bug. Telling people to create an issue on github, is already too hard for some. We should add a link in the overview-controlpanel to docs.plone.org. Make it easy to find articles on docs.plone.org, the search is not great.

Know the audience you are writing for. Are you targeting end-users or developers. Don't assume readers have your skill set.

Always write a readme, even if it is not perfect. At least it is a starting point, and people can help you out.

Don't postpone writing docs, just add a few lines when you are coding.

Include docs testing in your CI setup. If you want to, you can even include automatic screen shots.

Configure your editor. There are plugins for editors, to check pep8, pyflakes, but also docs: ReStructuredText, spell check, maybe link checkers.

Use a git hook to check your documentation when you commit. Look at mr.butler, mr.docs. mr.docs is a Docker image with the same Sphinx setup that we use for docs.plone.org.

There is not one solution to fix it all.

Let's have an open space about improving the situation. Maybe a QA team?

And: docs or it didn't happen!

Questions?

The English of the documentation is sometimes really clear, and sometimes really complex. Simple is better, you may not even need translation then.

The new patterns have tested documentation in their Javascript. I don't think this gets included in docs.plone.org? No, it is not nearly enough. Nathan has written blog posts. The mockup documentation is not maintainable, at least it cannot be included currently, we need ReStructuredText. We are getting there, but the Javascript parts need much more work.

Patternslib has its own documentation. Do we just link to this, or do we want to include it? If it is in core, we want to include it. Patternslib documentation is in MarkDown, but that will work fine. We will have to try it and think what is in the long run most suited.

[I missed the first lightning talks.]

bin/hostout app01 docker