Last year I laid all sorts of nice plans (in my yearly personal develoment
plan)
but it all got laid to waste by a full change of (company) direction. Poof
went my plans. Plans for my own direction, plans for teamwork, plans for lots
of improvements. That's life. And such a problematic year gives its own
opportunities for learning and improvement.
As an example of such an opportunity: somewhere in october I was quite
stressed.
I didn't agree with the direction we were taking. It was starting to take its
toll on my health (mostly my back muscles acting up). Now... how to get out of
it? My recipe was to tell it to my direct boss and to get his permission to
take a couple of afternoons off to cycle a bit. Of course I got the permission
(that's the good thing about where I work!) and spend some 6 afternoons
cycling within two weeks time.
That really helped clear my mind (and my body). And at about the same time I
started an all-new internal project: a relatively straightforward internal
Django timekeeping app. Lots of work and lots of tweaking, but at least a
stressless productive programming assignment. Just coming up with a good
architecture, a good css layout, some nice and proper code and yeah, I was
happy again.
So... I think the lesson for me is that giving early feedback is important.
And the company earned some bonus points for actually being happy with me for
telling about my problems and coming up myself with a simple way to fix/reduce
it.
I don't exactly remember my reply, but it came down to "no", I
wasn't aware of any problems with it. Just the next day however, at
the sprints, I encountered a fundamental problem with regards to
collaborative editing of Rich Text through TinyMCE.
The problem lies in the fact that the
Diff-Match-Patch
is meant for plain text only. It has no concept of structured
content such as the a DOM-tree and can therefore create problems
when one tries to patch HTML content. It seems that mainly two
things can happen. Firstly, because your HTML nodes are not atomic,
they can be broken by the patching algorithm inserting text inside
them. Secondly, the patching can result in HTML nodes appearing in
the wrong order. (See what the author of Diff-Match-Patch had to
say about it on
the project page
and his excellent
explanation on stackoverflow).
Ideally one should use a patching algorithm that is tree-based and
which will maintain the integrity of the DOM-tree. When using the
plain text algorithm, there are however two approaches to try and
mitigate the above issues.
Strip all HTML tags, diff-match-patch the resulting plain text
and re-insert the original tags.
Replace all tags with Unicode chars outside of the used range
(thereby making them atomic), apply diff-match-patch and then
restore all tags.
The problem with the first approach is that your markup is now
immutable. So any markup changes a collaborator makes (e.g bold,
italics, heading or other formatting) will be discarded. I think
this is a dealbreaker and rules out this approach.
The problem with the second approach is that even though your HTML
nodes are now atomic, they might still appear in the wrong order.
Because of this, I think trying to do collaborative editing via an
HTML-based Wysiwyg editor is not the way to go. Instead, a
Markdown editor (such as
Epic Editor) seems to be the way to go.
Markdown is plain text, and therefore doesn't suffer from the
problems mentioned above.
I think I have created the impression at the Plone conf that
production-ready collaborative editing via TinyMCE will soon be a
reality, and for that I would like to apologize.
I have to tell you a boring story about test isolation. On my flight back home from the Cathedral Sprint, I just solved an issue which stole half of the last sprint day's afternoon for Philip Bauer and me. Maybe it prevents someone else of having the same trouble.
plone.app.event's tests from the 1.1.x branch were failing for Plone 4.3 but not Plone 5 for absolutely no obvious reason. They were running for 4.3 some time before! We had some isolation problems - content created by test setup methods was still available for after test teardown, so succeeding test methods failed when trying to create these contents again. That problem persisted, even after Philip and I were changing all tests to use a functional test layer for better isolation.
After long investigations without success, finally a exclusion method helped me. I copied the source directory from the 1.0.x branch into the 1.1.x branch, so that I could easily restore individual files to their newer version with `$ git checkout`. I did that from top-down for the files `$ git status` listed up, running the whole test suite after each checkout.
Luckily I hadn't go down very far until until I found the guilty test. It was a test which was apparently copied over from somewhere else (I guess CMFPlone) which used ancient test frameworks - Products.PloneTestCase, Testing.ZopeTestCase and an installation of a Product I never heard about before: "Transience". See this commit for what has changed.
So, if you encounter strange test behavior which you cannot explain, look at your test setup and if some of your tests use old testing frameworks. Maybe this helps...
My attendance of this year's Cathedral Sprint was a must. Four (!) years ago, at the Cathedral Sprint 2010, plone.app.event was born (see Andreas Jung's blog article for more info). Soon I've become the maintainer of it, because I cared. Since then, I worked on this addon at nearly every other conference or sprint I attended. Which were quite some, let me see: Cathedral Sprint 2010, Buschenschanksprint 2010, DZUG Tagung Sprint 2010, Plone Conference Bristol 2010, Artsprint 2011, Buschenschanksprint 2011, Barsprint Ljubljana 2011 (that were Rok Garbas and me), Artsprint 2012, Buschenschanksprint 2012, Plone Conference Arnhem 2012, Artsprint 2013, Buschenschanksprint 2013, Arnhemsprint 2013 and finally Cathedral Sprint 2014. So 14 all together. Together with the Snow Sprint 2007 with Lennart Regebro, where I got involved into Plone calendaring, that would even be 15. We use plone.app.event in Production since some years, which - together with the in-production-usage by other companies - brought the package significantly forward. plone.app.event was finally included in Plone-Core sometime in 2013 and the so-called final release was cut just before the Arnhemsprint on 6th November 2013. A lot has been done since my last article on plone.app.event in 2011. Software takes time to mature and I'm happy with it's state.
The last month was quite busy with client projects and I couldn't review and merge any pull-requests (#132, #129, #127, #123, #119). So that was my first task at the Cathedral Sprint, together with a plone.app.event 1.0.5 release and a 1.0.6 release some hours later.
The next pull requests were already done at the sprint by Bo Simonsen (Rewrite of the portlets to make use of z3c.form instead of zope.formlib. Pull-Req #135) and Kees Hink (Removal of semantically incorrect dl-dt-dd from (all) portlets. Pull-Req #146). These pull requests have to wait until they are reviewed and merged together with corresponding pull-requests for other packages. I'm looking forward for having these merged!
Wolfgang Thomas did a great job in fixing some oddities in plone.app.event's Dexterity behaviors. First, he changed the behaviors from annotation to attribute storage (which makes easier access to behavior properties and less ZODB payload. Pull-Req #136), then he changed the recurrence attribute's RecurrenceField (which was introduced for easier overloading on a per field-type basis, but that turned out to be useless) back to a plain zope.schema Text field (Pull-Req #137) and then he removed the IEventSummary behavior, which is replaced by the IRichText behavior from plone.app.contenttypes (Pull-Req #140). He also provided migrations and migration-tests. Then Wolfgang started to integrate a timezone selection and a mail settings field in the @@plone-addsite view, which is responsible for creating a Plone site. This way, we avoid unset timezones, which would lead to wrong time calculations when creating events. He could not finish that work because of some dependencies on ongoing work for unifying the portal control panels into Products.CMFPlone, which also includes the removal of registry setting prefixes. Wolfgang was a great help!
I was fixing an old issue with the event view. Event views always had to iterate the whole recurrence set just to display the last recurrence. This could have been a potential bottleneck on large recurrence sets. However, I fixed this issue #60 and now the first 6 occurrences are displayed with a message on the number of remaining recurrences, if there are any (which I query directly from the index). Besides of being faster, this is also much more useful. I also changed the event view's detail table to a definition list, which is semantically more sense full and less problematic for accessibility (#140).
Then I added a robot test which covers creation and display of a Dexterity based event. Besides of raising test coverage and also covering JavaScript based functionality, the main purpose was to be broken by plone.app.widgets related merges. That way I could ensure, that the plone.app.widgets integration doesn't lead to regressions in functionality.
Now, in the week after the sprint, I released a 1.1b1 version (without plone.app.widgets integration). The releases are coordinated with Philip Bauer, who is maintaining plone.app.contenttypes. His 1.1b1 release will follow soon. plone.app.event also got cleaned up branches - there were too many before. We now have the master branch with plone.app.widgets integration, a 1.1.x branch from which the last release was cut (Plone 4.3+ compatible), a 1.0.x branch with Plone 4.2+ compatibility and some feature branches which will lead to pull requests once they are finished.
The plone.app.event improvements were relatively small steps compared to what happened in the other teams. I'm really happy to see David Glick merged the Barceloneta-Theme and plone.app.widgets PLIP into Plone-Core, with all tests going green! This was the main goal of the sprint and I'm looking forward to work with this. Altogether Plone 5 made a big leap forward! I'm looking forward for the next Plone 5 dedicated sprint I'm attending, the Wine and Beer Sprint in Munich next month.
Finally I have to thank Timo Stollenwerk for the excellent sprint organisation and the GfU Cyrus AG for hosting this sprint for the second time and covering lunches, snacks, drinks and even accommodation for invited sprinters! Without this help, the sprint wouldn't have been possible.
Again I have to say: this was one of the most productive sprints I've been at.
Here is a brief tutorial how to check if your domain’s SPF and DKIM spam protection measurements are configured correctly. These domain name records are used to prevent spammers sending email and forging the sender’s address to be your domain.
This blog discussed only about testing there measurements; there are plenty of online tutorials available how to setup the protection for your domain.
1. Checking your domain’s SPF and DKIM records, SpamAssassin score
You can check DKIM signature online using this great emailtest service by Brandon Checketts. You will get a random email address where you send in a test email and the results of the email delivery are shown online.
port25 solutions provides an alternative service where you can send inbound email and it will return the SPF and DKIM verification status back in the return message. Getting a reply from this service takes like 30 minutes.
Below is an example how to send out a test email from Django shell. Use the related method of your web framework sending out the test email. For plain UNIX you can always use command line mail command to send out a test email from your shell.
from django.core.mail import send_mail
send_mail("test subject", "test message", from_email="mikko@example.com", recipient_list=["example@www.brandonchecketts.com"])
Then you see the output online on the site above, like SpamAssassin results:
SpamAssassin Score: -2.011
Message is NOT marked as spam
Points breakdown:
-0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, low
trust
[198.2.128.3 listed in list.dnswl.org]
-0.0 SPF_HELO_PASS SPF: HELO matches SPF record
-0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay
domain
-0.0 SPF_PASS SPF: sender matches SPF record
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's
domain
0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
0.0 MSGID_FROM_MTA_HEADER Message-Id was added by a relay
Here is yet another SPF Policy Tester where you can enter your SMTP server IP address and domain name.
2. Testing forged email send
You also need to test that email receivers (GMail, Yahoo, HotMail) actually filter out messages based on the SPF test. This can be easily done by sending forged emails using the SMTP server of your local ISP. Telnet to SMTP server directly and then type out SMTP commands by hand.
Here is an example SMTP chat with a Finnish ISP’s outgoing SMTP server.
telnet posti.example.com 25
EHLO mikkos-imac
MAIL FROM: <mikko@sender.example.com>
RCPT TO: <mikko9999999@gmail.com>
DATA
test
.
3. GMail and SPF records
GMail does not seem to take spam decisions solely on either SPF hardfail -all or softfail ~all test. Thus, it seems to be very difficult to make GMail to discard spam and phishing attacks send in your domain’s name. Here is an example message from the above, as received by a GMail, which is delivered to inbox regardless of the domain SPF settings.
Delivered-To: mikko@receiver.example.com
Received: by 10.14.22.4 with SMTP id s4csp70004ees;
Wed, 19 Feb 2014 00:49:41 -0800 (PST)
X-Received: by 10.43.65.145 with SMTP id xm17mr25876368icb.35.1392799780856;
Wed, 19 Feb 2014 00:49:40 -0800 (PST)
Return-Path: <mikko@sender.example.com>
Received: from gw03.mail.visulahti.fi (gw03.mail.visulahti.fi. [1.1.1.1])
by mx.google.com with ESMTPS id sc10si873225igb.31.2014.02.19.00.49.37
for <mikko@receiver.example.com>
(version=TLSv1 cipher=RC4-SHA bits=128/128);
Wed, 19 Feb 2014 00:49:40 -0800 (PST)
Received-SPF: softfail (google.com: domain of transitioning mikko@sender.example.com does not designate 1.1.1.1 as permitted sender) client-ip=1.1.1.1;
Authentication-Results: mx.google.com;
spf=softfail (google.com: domain of transitioning mikko@sender.example.com does not designate 1.1.1.1 as permitted sender) smtp.mail=mikko@sender.example.com
Received: from mikkos-imac (a91-154-153-67.pulina-laajakaista.fi [1.1.1.2])
by gw03.mail.visulahti.fi (Postfix) with ESMTP id D639D216ADF
for <mikko@receiver.example.com>; Wed, 19 Feb 2014 10:49:13 +0200 (EET)
Message-Id: <20140219084922.D639D216ADF@gw03.mail.visulahti.fi>
Date: Wed, 19 Feb 2014 10:49:13 +0200 (EET)
From: mikko@sender.example.com
To: undisclosed-recipients:;
Test mail
If anyone knows how to make GMail to honor SPF, or to ignore forged sender messages, please let me know.
I'm a bit late, but yes, I had a great time at last week's Plone sprints. At first I flew to Amsterdam Stroopwafel Sprint for the weekend, and then continued by train to Cologne Cathedral Sprint 2014 for the week. Met old friends, got new ones, eat well and code a quite lot. Special thanks go to Sven for both organizing the Amsterdam sprint and hosting me there, Clean Clothes Campaign for providing the sprint facilities, Timo for organizing the Cologne sprint, GFU Cyrus AG for hosting the sprint and, of course, my employer, University of Jyväskylä, for allowing me to attend the sprints.
For the Sunday at Stroopwafel Sprint I paired with Giacomo to combine his work for making our end user and developer documentation translatable using Transifex with mine for translating also the screenshots. We ended up with a special buildout named papyrus.
So, Papyrus is a buildout for translating Plone-specific documentation into multiple languages. It's simple a buildout and a makefile to:
Build Sphinx-documentation with embedded Robot Framework-scripted Selenium-powered screenshots in multiple languages.
Extract translatable strings to gettext-POT-files for both translating Sphinx-documentation and embedded screenshots.
Push translatable files to Transifex and pull the translations form to build the local documentation.
Papyrus is a buildout for now, but it could be refactored into a recipe later. Anyway, it's designed to be separate tool bundle from the actual documentation to make it reusable by any Plone-related documentation later.
Currently, there's an example Travis-CI-configuration to build the current collective.usermanual in English and Italian. Unfortunately, there's not much translated yet, so there's sure a lot of work left for the next documentation sprints.
plone.themepreview
Image may be NSFW. Clik here to view.
During Cathedral Sprint 2014 I spent a lot of time to help the other sprinters with various Robot Framework and acceptance testing, but also made a small contribution for Theme and QA teams by recycling Timo's old theme screenshot suite into a reusable Sphinx-documentation called plone.themepreview.
plone.themepreview is a pre-written, Robot Framework and Selenium-powered, Sphinx-documentation with a lot of scripted screenshots for a Plone site with some client specific configuration – usually just a custom theme.
In other words, plone.themepreview comes with a Sphinx-scripts, which should should be able to launch a Plone sandbox with your theme and make a preview out of it to make it easier to evaluate if all the normal Plone use-cases have been covered by your brand new theme.
Of course, the current set of screenshots is not perfect, but if you think so, please, contribute! Just fork the project, do a pull request, and Travis-CI tell me if the changes are safe to merge. And if you have any questions, just file new issues!
Probably, also plone.themepreview could somehow be turned into a recipe at the end, but for now it seems to be easiest to just clone it and its build with theme specific configuration.
If I had just 20 minutes with a Plone newbie, eager to become a competent Plone developer, this is the advice that I would give to them today. I strongly believe that after the release of Plone 5, later this year, the specifics of this advice will be slightly different as projects like Mockup become the standard, but hopefully the general advice will hold true.
I'm assuming that you've already
Until now converting Plone into a bunch of HTML, CSS and JS files was a work to be done with wget or curl. You can now use stxnext.staticdeployment product, a handy product to convert your Plone site into static files. And even works with Dexterity. The "Plone Big Green Button" as described by Martin Aspeli, is a reality.
One of my latests articles was about HTML 5 Canvas and Webcam integration and in the same article I put all together in a Plone add-on for integrating portrait changes with Webcam.
Recently, following a retweet of one of the cool guys I follow on Twitter, I randomly hit an article that talk about a game that integrate user's webcam as a controller (unluckily I lost the original link, neither I remember the programming language used. The article were also generally introducing face recognition, and this captured my attention. I asked myself how cool could be getting a face recognition feature with Python. It this something possible to do? Probably not so easily...
Introducing OpenCV
...or not? If you look for "face recognition" and "Python" on Google you'll always get references to OpenCV, the Open Source Computer Vision Python library. I simply scratched the surface of this huge piece of technology as I'm totally a newbie about computer vision. What I get is that the library can really do a lot of stuff, and it's well documented.
Let me introduce some very-general information.
To use OpenCV for detecting faces on an image you must understand the difference between "face recognition" (find a know face on image or video) and "face detection" (find a face, in general). Obviously the second is a simpler task. Why? Because whatever is your task, OpenCV must be trained to find your target. For not simple task like face recognition you can (for example) train the software by providing a set of images where the object face can be found, and the result of the train is an XML file. After that you can you this file to implement something like Picasa, iPhoto or Facebook are doing when you submit new photos.
With face detection things are simpler because you can find one of those XML file online, already generated for you.
Another important informations about the library: recently a deep API changes has been performed so a lot of examples you can find online are broken (or must be fixed).
Finally, when you are able to do so, prefer the use of cv2 library instead of cv. They are more or less the same library but (for what I understand) cv2 is faster because is based on numpy, so C-compiled code.
Going back to what i did: I focused on face detection. Let see how.
Applying face detection to Plone (yes, I said it)
Meanwhile I were also fixing some minor issues in collective.takeaportrait. A new minor feature is the possibility to move the viewfinder by using mouse drag&drop (because must be forced to be in the middle of the screen is not comfortable). Here came The Idea: how about a viewfinder that automatically center onto the face captured by the webcam?
Here my wish list:
JavaScript check for server side availability of face detection feature (just because OpenCV is not a simple library to be installed... and let me be honest: this is a cool feature, but not really useful)
With a not-so-long delay the whole Webcam image take by the canvas is sent to the server for a face detection view
OpenCV on the server perform the face detection
If a face is found, a rect is sent back to the JavaScript callback
The viewfinder is centered on the face using the feature already implemented for drag&drop
The use of Plone here it's a bit unnatural but It was the simplest environment for my experiment because of the work already done with the webcam in the last article. Apart Plone itself I hope you'll the general idea: how simple can be the browser/webcam/face-detection integration, whatever will be your back-end Python framework.
As you can suppose, the experiment was a success!
Introducing collective.takeaportrait face detection feature
I don't think I can add more useful details. Let's see the video!
Image may be NSFW. Clik here to view.Often people (mostly Gogo) are running into massive problems with Plone buildouts. Problem is here not Plone, but setuptools: it was forked into distribute and setuptools, then the reunion of both, the further development. Future looks good for setuptools. But ghosts of the past are still hunting us.
So in short there are only a few things todo:
use Python 2.7, with virtualenv >=1.9.1 with parameter --no-setuptools
Last time I talked about how I went back to the center of the Zope
project. Over the course of the year following, we managed to refactor
the Zope Toolkit, clean up the dependency structure, and we could drop
many of its unwanted dependencies.
I had hoped that with new leadership, a steering group, we could also
reinvigorate the Zope project itself. Could we get together and do
exciting new things again?
The answer was no.
Chris McDonough
In late march 2009, I visited PyCon, in Chicago. There I had a
conversation with Chris McDonough, who was working on what was to
become the Pyramid web framework. He and I had a conversation about
the dependency cleanup project I had started and that had been making
waves on the zope-dev mailing list. Some beer was involved, and some
miscommunication. Chris was skeptical that the cleanup project would
succeed within the year, which confused me a bit, as we had already
made a lot of progress.
But I was talking about cleaning up circular dependencies and the
ability to lose a lot of the code. Chris was talking about making
libraries with a clear purpose and documentation.
And while we got better dependencies at the end of the year, I
failed.
Burning out on Zope
The details of the straw that broke the camel's back (though I hardly
have the stamina of a camel) are immaterial. Suffice it to say that
when practical disagreements happened our steering group did not
function.
So in early 2010 I realized I was putting more into the project than I
was getting out of it. I was running out of steam. It was costing me
emotionally. So I stopped going to the Zope mailing lists. While we
had made progress on matters where there was consensus, maintenance
seemed the only consensus that could still be found.
Consensus on the boring stuff is not enough when the web is
changing. And the web was changing, as it always is. A lot of the
innovation on the web was happening on the client-side, in JavaScript,
but Zope had no client-side story. People had moved into different
directions, and the community had fractured.
The dependency cleanup was just about the only progress being made
-- what about my personal goals? Where was the creativity, the getting
together to do new interesting things? It wasn't there. Instead we had
a bunch of opinionated people who couldn't agree enough to get
anything but basic maintenance work done, and stumbled doing even
that.
I'd been involved in Zope heavily, also serving on the Zope Foundation
board for some years, including as its chairman. I decided I had to
pull back from all of it.
The Zope Summit
In September 2010 I found myself at a Zope Developer Summit, which had
been organized in part because of my urging. I had been heavily
invested in Zope, and had been for more than 10 years. I had used
Zope, benefited from Zope, contributed to Zope, redefined Zope and
built on Zope. I had learned from Zope. I had been first board member
and then chairman of the Zope Foundation.
I had hoped that a summit could get things moving again. Talk about
cool new things that we might do together.
Zope was in trouble. The codebase would live on. It still does. But it
is in maintenance mode - it doesn't do much that is new.
I came to the Zope Summit with a gloomy heart. That did not help my
mood at the summit -- sorry.
I think most of us left the summit with the feeling not enough had
been accomplished. The future of Zope was disappearing. Zope had lost
its power to adapt. The web changes, but Zope didn't anymore.
This was the end of Zope, for me. I still use it in the shape of Grok
to this day, but it is not the same.
Life After Zope
But this is not the end. The code still continues, and is being used,
though is mostly in maintenance mode.
And there is life after Zope. Next I will talk a bit about what came
after.
Here I am, sitting in my apartment in Ferrara, after a travel that I will remember for all my life. These last 10 days have been an amazing experience of work and life with the Plone community. Thanks to RedTurtle, I just had the chance to attend both the documentation sprint in Amsterdam and the Cathedral Sprint in Cologne. I have to say that these two sprints have been very much productive.
Stroopwafel Sprint - Amsterdam - February 7th-9th
Image may be NSFW. Clik here to view.I left my hometown on Thursday 6th, destination Amsterdam, for the Stroopwafel sprint organized by Sven Strack and Paul Roeland.
The organizers and me too, were very surprised by the unexpected partecipation to the sprint, since normally the documentation is not the "cool sprint" everybody wants to attend. Instead the participants were numerous, I think more than 10 people from many different countries. We started on Friday 7th with a whole day of brain storming about what the current status of the Plone's documentation is, what we wanted to achieve during the sprint and what was our final goal for the deadline of Plone 5 release (and beyond). With the help of Sisi Nutt we tried to get through the current documentation with different prospectives, trying to follow the flow of a newbie that will try to get into Plone.
Then, we started filling post-its, and with them we started filling the wall. We ended up with 3 walls full of post-its with ideas and notes. We found the gaps in the flow of informations, we found the fields that needs to be cleared better, we found the sources where some more documentation can be found. The goal was clear, we need one single location for all the documentation and it has to be:
clear and easy to read
useful
translated
up to date and therefore versioned
easily maintainable
cool (why not?!).
We also tried to spot the targets (end users, designers, developers, integrators and so on) and to organize the documentation to be effective for all of them. The original idea for the sprint was just to organize the ideas and the tasks and then we would have started the "real work" in Munich at the second documentation sprint, but we ended up also doing some technical preparation.
The result is that we decided to separate the documentation itself that will be written in ReST from the buildout that will build it. This approach will let us have branches on github for each Plone's version (and subversion) and keep a single buildout for all of them. We started to work on the buildout that will manage all this: https://github.com/plone/papyrus
Together with Asko Soukka, we also added the auto-generation of the screenshots. This will not only permit to have the screenshots with many different Plone themes, but also they will be automatically translated. Amazing!
In order to get all this, Sven and me, joined the AI-team and we'll continue to work on the technical part during these days. Next stop: the documentation sprint in Munich. There, with the help of some students from the university, we'll do some real work.
Cathedral Sprint - Cologne - February 10th-16th
Image may be NSFW. Clik here to view.On Sunday 9th, many sprinters in Amsterdam left for Cologne where we joined, along with my colleagues Alessandro Pisa and Andrew Mleczko, the Cathedral Sprint. The same night we also met many other plonistas that were already sprinting in their hotel rooms (even though the sprint would have start the day after...man, that's the enthusiasm behind the Plone community!).
On Monday 10th the sprint began. The excellent organizer of the sprint, Timo Stollenwerk, set up everything during the kickoff meeting for being productive and effective. We decided to have these tracks:
I decided to join the theme team since I have now some experience in that field. We improved very much the theme, for example now the edit bar is a vertical, black, elegant bar, which includes also the personal menu and the portlet management. The new theme will be also responsive out of the box.
We also focused on improving the user experience, for example in the next Plone's release all the viewlets in the footer will be replaced by portlets which will let the user customize them more easily.
Paul Roeland also worked together with both the theme team and the QA team to assure that the next Plone UI will be perfectly accessible and will follow the latest wcag guidelines.
The javascript team, where Alessandro and Andrew joined, also managed to do many tasks. They for example moved the new theme from Twitter bootstrap version 3 to version 3.1, they improved and fixed the javascript tests for the new widgets and also wrote new widgets (i.e. there will be an upload widget which will let the use upload multiple files at once). The next Plone release will also feature TinyMCE 4 which is a great improvement in usability and accessibility.
These teams mainly focused on fixing tests failures and write new tests. At the end of the week, thanks to them, we had a green build in the Plone5 tests. Yeaaah!
This team, where Philip Bauer and Johannes Raggam worked, improved and almost completed the migrations from archetypes to dexterity content types, which will be provided in the next release.
This team worked on improving the look and feel of the next revision of the plone.org site. They focused mainly on the author page, that will feature many statistics of the developer like the number of commits from github, the number of answers on stackoverflow, the number of tweets and so on. This will expose the reliability and credibility of the developers.
The end
At the end of the week we almost managed to achieve the initial goal: having a Plone 5 alpha release. But we were not that far, since in these days the main PLIPS have been merged with all the tests passing. The alpha release is right after the corner, stay tuned!
At the end, I would really like to thanks the organizer of the sprint, Timo Stollenwerk, which has done an amazing work in bringing there more than 30 plonistas from all around the world. I also want to thanks the center GFU which kindly provided the location, the catering and the food.
I felt like I was racing on the Morepath today. My goal was to see
how to integrate Morepath with a database. To make this goal
practical, I looked into integrating Morepath with SQLAlchemy. To go
faster, I borrowed ideas and code liberally from Pyramid.
Tweens
This morning I borrowed the idea of tweens from Pyramid. A tween is
basically much like a WSGI framework component, but one that knows about
the web framework -- it gets the web framework's request, and it can
send the web framework's response, among other things. Now you can write
this with Morepath:
What happens now? Morepath will automatically commit the transaction
if there is no error (like a 500 error, say).
This means that Morepath now has integration with databases that use
the transaction module, such as the ZODB and also SQLALchemy (using
zope.sqlalchemy for transaction integration).
[update after someone blindly complains after seeing the word "zope"
in a package name... Please don't do that.]
Moreover, you can use multiple such databases in the same
application. You can modify the ZODB and a relational database in an
application, and be secure that if anything fails during the request
handling, none of the databases will changed -- both transactions
will be aborted. There's a lot of goodness in the transaction module.
Morepath settings infrastructure
It turns out pyramid_tm is configurable in various ways. It allows you
to set the number of attempts it will try to commit in the face of
conflicts, for instance. To support this, I had to build Morepath's
settings infrastructure; just the part where you can have settings at
all, not loading them from a config file -- that's for later.
Here's an example of the settings for the transaction app in
more.transaction:
These are the defaults defined by more.transaction, but they can be
easily overridden in your own app (by writing the same code as above
with different values for the configuration).
When I started to write Morepath's settings infrastructure I wrote
quite a bit of code involving combining and extending settings, only
to throw it away by replacing it with much shorter code that builds on
Morepath's config engine that I already use for its other
directives. Nice!
morepath_sqlalchemy
Now that I had all the pieces I needed to put them together to
demonstrate SQLAlchemy integration: morepath_sqlalchemy.
This uses more.transaction, zope.sqlalchemy and SQLAlchemy
together. It's all done here, but I'll summarize the important bits
of integration here:
fromsqlalchemy.ormimportscoped_session,sessionmaker# SQLAlchemy sessionSession=scoped_session(sessionmaker())fromzope.sqlalchemyimportregister# register session with transaction moduleregister(Session)importmorepathfrommore.transactionimporttransaction_app# create our app, extending transaction_app.# this means we get Morepath transaction integration# and default settings for it.app=morepath.App(extends=[transaction_app])
Quite a day
It's all still rough, needs polishing and documenting, but the
foundations are now there. Quite the day of coding! I couldn't have
done it without the Pyramid project from which I could borrow many an
idea and piece of code.
Now I know Morepath can be integrated with any kind of database -- if
transaction module integration is there, like for SQLAlchemy and the
ZODB, it is very easy: just use more.transaction. But even if not,
it should now be possible to write a tween that does the trick.
Tweens allow other neat things too: I think I saw Pyramid's custom
error view system is based on a tween, and I still need custom error
views in Morepath...
Interested? Hope to hear from you! Join #morepath on freenode IRC,
drop me an email, or leave a comment on the issue tracker.
I’m starting to get excited about the 2014 Plone Emerald Sprint which is happening in a couple weeks on lovely Whidbey Island near Seattle.
The focus of the sprint will be doing some work on Plone’s user membership features in preparation for Plone 5. Now, the sprint will be more effective if we all (the sprinters, and the wider community) have an idea of what needs to happen before we arrive. To that end, I just spent some time taking screenshots of the current state of things in Plone 5 (yeah yeah, I know, it’s a job for a robot) and making lists of what I think should happen.
So here goes. Look, there are pictures!
Login form
Image may be NSFW. Clik here to view.
This is currently implemented as a bunch of old CMF skin templates and scripts. Should be rewritten using z3c.form. Whoever works on this should start by reviewing the work Gil Forcada already did (https://github.com/gforcada/login_forms).
Dialog is missing a title
Say “username” instead of “login name,” for consistency
Fix bug where a user logs out and then logs in again, and is taken to the logged out page.
Password reset
Image may be NSFW. Clik here to view.
This is also old code using CMF templates. Should be rewritten using z3c.form
Say “username” instead of “user name,” for consistency
Improve wording of the email that’s sent to the user
Liz, you had some bugs related to this, right?
After user sets their password, log them in immediately (?)
Would be nice to show a checkbox in real time for whether the new password meets the criteria.
Registration form
Image may be NSFW. Clik here to view.
Eric Brehault & co. have done some good work on PLIP 13350, which aims to make it possible to edit the member data schema through the web with the Dexterity schema editor — so you can collect whatever information you want from users! There are a few tasks which need to happen to finish getting this ready to integrate.
Personal information form
Image may be NSFW. Clik here to view.
This is the form for editing your member info once you’ve already registered. It has already been converted to z3c.form. Integrating PLIP 13350 will also affect it.
Author page
Image may be NSFW. Clik here to view.
This is linked from the byline of pages and is the closest thing we currently have to a public user profile. It would be nice to make it possible to select fields from the member data schema to be shown here.
It would also be nice to add an ‘edit profile’ button here, for users with sufficient permission.
Preferences Form
Image may be NSFW. Clik here to view.
I always get confused between this one and the Personal Information form. This has settings that affect one’s use of the CMS, as opposed to information about a person.
I think it would make sense to merge this into the Personal Information form, as a separate fieldset.
We should remove the per-user external edit preference.
We should remove the per-user ‘edit short name’ preference and instead always show the short name field on settings tab of edit forms for anyone who has permission to rename.
User/groups control panel
Image may be NSFW. Clik here to view.
Plenty of room for UI improvement here
Hide control panel navigation portlet, or find a better way to show it that doesn’t eat horizontal space.
Need to fix the navigation between users and groups
The options on the ‘settings’ and ‘member registration’ tabs should be moved to the Security control panel, which should be renamed to something like ‘Member registration’
Can we automatically determine when to use the ‘many users’ and ‘many groups’ flags?
It would be nice to add a way to mark a user as “inactive” so they can’t log in
It would be nice to show a user’s last login time
Sharing tab
Image may be NSFW. Clik here to view.
Plenty of room for UI improvement here
Folks at the Cathedral sprint recommended renaming this to “Grant access,” since “sharing” tends to mean social media these days.
Should this show in an overlay?
We should make it clearer that the columns are roles, not permissions, and what exactly they do.
Other topics
The above pages are the ones that I consider essential to have working well for a Plone 5 release. There are other possible topics related to Plone members, and if someone is super motivated to work on one of those, don’t let me stand in the way. Two that come to mind for me that are each pretty big projects on their own are:
It would be nice to add optional support for logging in via an external authentication source (such as google, facebook, etc)
It would be nice to modernize FacultyStaffDirectory, convert it to Dexterity, etc.
If you’ve made it this far and are excited about the possibility for improvements to Plone’s member infrastructure, but can’t make it to the sprint yourself, please consider donating to help cover the costs of the sprint.
So what happens? Instead of ending the traversal on the view the IPublishTraverse interface is found by the publisher. Now it looks for a publishTraverse method and calls it until the traversal stack is empty.
This week at the March Jamaica Plone Meetup will be focused on learning about using the Robot Framework. This is a video designed to get participants up to speed before the event.
I've also prepared some slides and a github gist
The Gist
Today I changed over Morepath to use WebOb instead of Werkzeug as
its request and response implementation.
Morepath is your friendly neighborhood Python web micro framework with
super powers.
In this post I'd like to explain what lead me there.
Having had now quite a bit of experience with both Werkzeug and WebOb
I will offer some points of comparison and feedback that may be useful
to improve Werkzeug and WebOb both, but I'll do that in a followup
post.
Performance Testing
Two weeks ago I gave a talk about Morepath in Singapore, for the
Python Singapore User Group. When preparing the talk I ran into a blog
post describing a performance comparison between web frameworks called
Python Fastest Web Framework.
Now Morepath is not striving to be Python's fastest web
framework. It's striving to be fast enough, and offer a lot of power
and flexibility to developers in a small package. Morepath offers some
special features such as linking and application reuse.
A performance comparison between web frameworks implies functional
equivalence between them, but they are not: some web frameworks like
Morepath have powers that others don't have. Using those powers may
allow you to build applications more quickly, and also organize them
in ways so that they are faster end than is easy to accomplish with
other, less versatile frameworks.
In addition we know that real world web applications typically have so
much overhead doing other things (such as dealing with databases) that
simple things like request handling are a minimal contribution to
performance in the end.
All that aside, I still wondered how Morepath did compared to other
web frameworks. Of course I did! It's nice to be able to say your web
framework is fast. More subtly benchmarking can also say something
about the amount of work a web framework does to serve a request, and
the less work, arguably the easier it is to understand the web
framework and to debug it.
So I plugged Morepath into the "hello world" page benchmark and found
Morepath was about as fast as Flask, but that Flask was one of the
slower of the lot compared:
msec here is the amount of milliseconds to run all 100,000
requests in the benchmark, rps is amount of requests per second,
tcalls is the total amount of function calls to handle a single
request, and funcs is the amount of different functions called
during request handling.
Morepath in this benchmark is about the same speed as Flask. Morepath
is slower at this benchmark than Django, Tornado. Pyramid does pretty
well too, which is not a surprise due to its focus on
performance. Morepath is hopelessly slower at this benchmark than
speed monsters such as Falcon or wheezy.web. wheezy.web, the web
framework by the author of the blog entry.
That evening when I gave my presentation someone actually referenced
this benchmark and wheezy.web and asked how fast is Morepath. Having
done my research I could answer him Morepath is about as fast as
Flask, and also gave the caveats concerning performance above.
Still, could Morepath not do better? Morepath is about as fast as
Flask in this benchmark. Perhaps the Werkzeug library that both
Morepath and Flask used was the common factor dragging performance
down?
wheezy.http
wheezy.web is fast, so I took a look at this. I discovered wheezy.web
is built on wheezy.http, which is a library that abstracts request
and response from WSGI much like WebOb and Werkzeug do.
After coming back from Singapore to the Netherlands, I looked up the
author of wheezy.web and wheezy.http up on IRC, and had a nice
conversation with him. He pointed out that his benchmarking system has
a knob that shows profiler statistics. I turned it on and this is what
I saw:
Lots of this stuff I recognize as the internals of Reg, the generic
function call library that Morepath is built on and that is already a
known candidate for optimization efforts, but that will have to wait
until later. We care about the request/response implementation now.
Werkzeug shows up twice in the top 10. First there's response object
generation:
This is an enormous amount of calls to isinstance(). I recognized
this as due to Werkzeug as the profile for Flask showed the exact same
number of calls (3100002), strongly suggesting Werkzeug as the
cause.
I bit the bullet and experimentally changed Morepath to use
wheezy.http as its request/response implementation instead of
Morepath. This caused the request/response implementation to
completely disappear from the top 10 most expensive functions. The
isinstance stuff was gone too.
Morepath was 47% faster on helloworld now than with Werkzeug!
Armin Ronacher responded to this result on Twitter, and said the
isinstance business is likely a performance regression due to
Python 3 compatibility in Werkzeug...
Switching to wheezy.http?
I wondered now whether I should switch Morepath to wheezy.http. It is
certainly an attractive library, along with some of the other wheezy.*
libraries.
My main trouble with it is that wheezy.http has seen much less
real-world battle testing than either Werkzeug or WebOb. Looking at
its source code the request and response wrappers were very simple
indeed, which made them a lot easier to read than the equivalent
implementations in Werkzeug and WebOb. That is certainly
attractive. But they also seemed to do rather little with
encodings. And later I heard from Chris McDonough that wheezy.http
will have trouble dealing with non-ascii URLs.
WebOb
There was an obvious candidate sitting around that I hadn't tried yet:
WebOb.
I had initially deliberately avoided using WebOb for Morepath for two
reasons:
when I had to do some other WSGI work I found that Werkzeug had a
nicer lower-level API exposed that let me work with raw WSGI better.
Pyramid is already using WebOb, and I figured since Morepath was
already similar enough to Pyramid anyway I could try Werkzeug for a
change. Perhaps using it would benefit Morepath in some ways I could
not foresee.
The second reason wasn't very good, except for one thing: I learned
more about Flask and could model aspects of the Morepath documentation
after it. Otherwise Werkzeug and WebOb are pretty interchangeable. And
I'm confident Morepath is different enough from Pyramid anyway.
Now I had a strong reason to try WebOb: performance. I know that Chris
McDonough had been working on WebOb a lot and that he cares a lot
about performance, so I figured I should give it a shot.
So I swapped in WebOb and tried the benchmark again. The first result
was disappointing:
WebOb's request.py __setattr__ was showing up at number 3. I
discovered that WebOb's request object has some magic that observes
attributes. I also discovered that WebOb has a BaseRequest that
doesn't include this magic.
WebOb: the results
So I tried things again using BaseRequest instead:
That makes Morepath 30% faster with WebOb than with Werkzeug and
faster than Flask.
Not as good as with wheezy.http, but using a much more battle-tested
framework, so not bad. This gets Morepath closer to Django and Tornado
at this. Once I optimize Reg I think I can get closer still.
Also see that the amount of functions called during a request dropped
from 314 to 245, and the amount of functions used drom and used has
dropped from 122 to 92.
Switching Morepath to WebOb
If I wanted to switch something as big as a request/response implementation,
now was the time: before a Morepath release. So I made the switch.
It wasn't difficult; the APIs are very similar. The most work was
actually porting the Morepath tests, but that got a lot easier once I
discovered the webobtoolkit library.
Another benefit of switching to WebOb is that is may eventually allow
more code sharing between the Morepath and Pyramid ecosystems. I
suspect the easier candidate for code sharing would be Tweens, as
Morepath and Pyramid now have the same basic Tween API.
Tomorrow I'll follow up this post with some feedback about Werkzeug
and WebOb in general.
El Salón de Arte Digital de Maracaibo se celebra anualmente con participantes locales y foráneos, con el objetivo de reunir y mostrar el trabajo creativo de todo el que desee participar que involucre la tecnología en su proceso creador o como plataforma. Cada año hay diversos participantes divididos en cuatro categorías: impreso (subdividido en fotografías e ilustraciones digitales y digitalizadas), videos (animación 2D, 3D, Flash y stop motion), música digital y multimedia. Además de una Charla-Foro durante los cuatro días del evento bajo el título Creatividad + Tecnología.
¿De dónde viene la idea de hacer un Salón de Arte Digital en Maracaibo?
Según Fernando Asián, Fundador del Salón de Arte Digital ”La idea nace de la necesidad. En Venezuela dejaron de celebrarse salones nacionales de arte desde 1989. Como artista plástico participé en muchos de ellos, siendo reconocido con premios como el Premio Emilio Boggio para Dibujo en el Salón Arturo Michelena, y el Premio Metro de Caracas para Dibujo, en el Salón Nacional de Artes Plásticas, 1989, entre otros. Deduje que la inevitable, y cada vez más presente, tecnología era un elemento aglutinador, o mejor, una excusa para convocar actividades artísticas que usan el formato digital en su realización. Esta incorporación del elemento tecnológico facilita la participación de cualquier género, lo que permite romper el paradigma del salón clásico. Por otra parte la herramienta digital no producirá un nuevo tipo de arte, pero favorece la interrelación entre géneros y facilita la producción de expresiones como el cine, la multimedia, etc…¨
Salón de Arte Digital, es un evento realizado por 10 ediciones, que reúne artistas multimedia de todo el mundo. Este edición ofrece nuevas tecnologías, nuevos medios en las artes y en las ciencias. Este evento se estará realizando desde el 10 al 28 de Marzo 2014.
Información básica
Lugar: PDVSA La Estancia Maracaibo. Ubicado en la Calle 77 (bulevar 5 de Julio), parroquia Olegario Villalobos del Municipio Maracaibo, al lado del Edificio Principal de Enelven.
Costo: ENTRADA TOTALMENTE LIBRE Y GRATUITO
Inscripciones: Desde el 5 de marzo de 2014, en la recepción de PDVSA La Estancia Maracaibo.
Cronograma de Software Libre en el Salón de Arte Digital de Maracaibo.
Primer día – 10 de marzo
Titulo: ¿Qué es el Software Libre?
Descripción: esta cátedra habla de Historia, Definición y Filosofía de The GNU Project, Free Software Foundation (FSF), Software Libre (Free Software), El Núcleo Linux, Código Abierto (Open Source), Open Source Initiative (OSI). Además trata sobre las Licencias, Distribuciones, Alternativas Libres, Comunidades de Software Libre, Ventajas, Desventajas, Situación Actual del SL en Vzla, Sitios en Internet.
Duración: 1 hora académica
Titulo: ¿Qué es Hardware Abierto?
Descripción: esta cátedra habla de la Problemática actual, Historia, Lee Felsenstein y el Homebrew Computer Club, FPGAs y Open Design Circuits. Definición y Filosofía según su naturaleza (Hardware estático y Hardware reconfigurable); y según su filosofía, Open Hardware, Open source hardware, Free hardware design, Libre hardware design, Free hardware. Además trata sobre el Licenciamiento, Comercialización, Modelos de intercambio, Proyectos, Comunidades de Hardware Abierto, Ventajas y Desventajas, Situación actual en Venezuela, Conclusiones, Sitios de Referencia.
Duración: 1 hora académica
Titulo: Del uso de la tecnología a la innovación tecnológica.
Descripción: esta cátedra habla de la experiencias de muchos activistas Venezolanos dentro de las comunidades de tecnologías libres y su formación socio política ante la tecnología y su impacto en la sociedad, formando a nuevos ciudadanos, estos actualmente no solo son activista en tecnologías que promuevan el conocimiento libre sino forman parte muchos movimientos culturales y sociales a nivel mundial, nacionalmente y regionalmente. Con esta experiencia se educa como involucrarse en estos movimientos.
Duración: 1 hora académica
Segundo día – 11 de marzo
Titulo: Distribuciones de Software Libre Venezolanas
Descripción: esta cátedra habla de Distribuciones GNU/Linux, ¡GNU/Linux a la medida!, Clasificación de Distribuciones, Historia en Árbol genealógico y Mapa mental de distros. Además las Implementaciones Venezolanas de distros como: HVLinux, Bluewall GNU/Linux, KnoppixMED, LinuxDoc, Latinux, GNU/Linux – Venezuela, Soliedelca, Lavicux, ULANIX y sus sabores, Kuntur y CANAIMA GNU/Linux, con algunas demostraciones y anexos las referencias a Sitios en Internet.
Duración: 1 hora académica
Titulo: Proyecto Canaima
Descripción: esta cátedra habla del proyecto Canaima surge como un proyecto para normalizar la migración a Software libre en al APN en Venezuela, en esta charla busca compartir las experiencia del proyecto Canaima globalmente no solo como un producto tecnológico sino como el estado ofreció la apertura a la comunidades de software libre para compartir saberes y ayudar a construir una base de socio productiva en tecnologías libres en Venezuela, aquí se comenta un poco de su historia, productos tecnológicos y organización social de la comunidad hasta el momento, encuentros comunitarios y recursos tecnológicos disponibles en redes sociales e Internet para colaborar virtualmente con el proyecto.
Duración: 1 hora académica
Titulo: Diseño Gráfico Digital en Software Libre
Descripción: esta cátedra habla de Alternativas Libres para Editor de gráficos; Editor de gráficos vectoriales; Editor de animación 2D; Editor de gráficos 3D; Editor de gráficos de post-producción; Editor de Fuentes; Maquetación y Publicación; Animaciones Web; Editores Web; con sus características técnicas y tabla de costos por licenciamiento de software. Además trata sobre Sitios en Internet y Demostraciones.
Duración: 1 hora académica
Tercer día – 12 de marzo
Titulo: Sistema de gestión de contenidos Plone
Descripción: esta cátedra habla de como crear facilmente usando el software de sistema de gestión de contenidos Plone ideal para creacion de Paginas Web
Plone es el mas longevo, poderoso, flexible, seguro y premiado sistema de gestión de contenido escrito en Python. Utilizado en sitios como Brasil.gov.br, CIA.gov y VTV.gob.ve, él es conocido por su seguridad e flexibilidad. En esta charla exploraremos, ¿cómo funciona él?, ¿cómo funciona su comunidad organizada?, y vemos como los usuarios exigentes con fechas límite de entrega le encanta Plone, ya que les permite no tener que inventar la rueda de nuevo cada vez que trabajan en la carga de contenidos en la Web.
Duración: 1 hora académica
Titulo: Traducción asistidas por computadoras en Software Libre
Descripción: esta cátedra habla de ¿Qué es la traducción?, describir los recursos del traductor, también como ha evolucionado hasta la actualidad la industria de la Lengua. Herramientas CAT (Computer-aided Translation), Memorias de Traducción. Se ofrece un análisis comparativo entre herramientas CAT privativas y libres donde se describen sus ventajas y desventajas, costo de licencias de software y soporte. También se ofrecen algunos recursos en la red y demostraciones.
Duración: 1 hora académica
Titulo: Montaje de un proyecto de Software Libre
Descripción: esta cátedra habla de ¿Cómo montar un proyecto de software libre?, lo que hay que hacer antes de comenzar el proyecto se explican los temas de: Lenguaje de programación, Plataforma, Especificaciones, Luego se debe definir algunas metodologías de desarrollos que se asemejen a SL y las herramientas de trabajo colaborativo para el desarrollo y control de gestión y proyectos para definir roles del proyecto, uso de recursos CVS, Lista de correos; tales como algunos recursos de la Web 2.0 disponibles para empezar a organizarse, y también sobre algunos FOSP-hostsites. Y una vez que esta listo la primera publicación se deben tomar en cuentas los aspectos legales, buenas prácticas al liberar, la publicidad del proyecto, el empaquetamiento y distribución (fuentes y binarios), el cuidado con la imagen, el soporte a usuarios: sitios de referencia, BTS, comunidades. Además trata sobre el Tiempo de dedicación al proyecto, Conclusiones, Referencias, Sitios en Internet, Demostraciones, Preguntas.
Duración: 1 hora académica
Del 13 al 18 de marzo
Actividades simultaneas de talleres de diseño gráfico y dibujo, exposición de arte. Mas información recepción de PDVSA La Estancia Maracaibo.
