Quantcast
Channel: Planet Plone - Where Developers And Integrators Write
Viewing all 3535 articles
Browse latest View live

Plone Emerald Sprint: Emerald Sprint design recap


Plone Emerald Sprint: Emerald Sprint was awesome

$
0
0

Hey all!

Back in California now but I was definitely having withdrawals as soon as I walked away from the #airportsprint. Let’s do this again soon! Anyway, I put together a fancy blog post about what I learned during the sprint. I’m pretty sure every time I talked to someone new, I picked up another great Plone tip, but I recapped the top three for starters. — https://medium.com/p/e7dc4757e4b4

Also, for posterity, here are the assets for both the login screen mock-ups and the Plone.com mock-ups. Feel free to remix ‘em if you want to! (Bonus: the Emerald Sprint logo is in there, too.) — https://www.dropbox.com/sh/c02o8j8rysrkxlh/XEYhGzOLiD 

 

Til next time,

Trish 

Steve McMahon: User Property Frolics on the Sound

$
0
0

Much is possible, with a little help from your friends -- and a few sprint sponsors.

The recent Emerald Sprint concentrated on user issues: how those pesky people who don't work at the command prompt register, login, logout and fill out their user profiles. With Ian Anderson and Luke Brannon, I worked on PLIP 13350, a proposal to make the fields that make up a user profile easy to extend and edit.

Dinner

Changing user profile fields is one of those long-term Plone annoyances. It's something that's very easy to do -- for an experienced integrator -- but completely opaque to new integrators and developers. Eric Brehault (ebrehault) and Mathieu Le Marec - Pasquet (kiorky) submitted a plan to clean this up with PLIP 13350 using something like the Dexterity field editor. They also laid the implementation groundwork, actually using Dexterity's schema and schema editor.

Ian, Luke and I took on getting this ready for Plone 5. We also wanted to reconcile Eric and Mathieu's work with the one of the big ideas of the Emerald Sprint: that user profile fields be distributed through registration, mandatory profile updating and profile editing.

Luke and I scored big by getting to work with Ian. Re-purposing Dexterity's schema editor to work outside of Dexterity's content type configlet is a mind-bending bit of manipulation of z3c.form, plone.schemaeditor, plone.supermodel and plone.app.z3cform. David Glick also added in another ingredient: new implementations of Edit and URI zope schema fields in plone.schema. All this is the sort of thing I can usually figure out in a week or two, but this was a three-day sprint. Ian, though, got the hard part figured out pretty quickly, and Luke and I were able to follow him around writing tests and cleaning up details.

We didn't finish, but by sprint's end it was pretty much working. Ian had even consolidated all the fields into a single schema with field-specific flags to indicate which forms the belonged on. David jumped in from across the table to add a "these fields may not be deleted" list to the schemaeditor, and Ian was able to use it to protect fullname and email address fields -- the ones we need for Plone basics and can't allow a site admin to delete.

All of this was the kind of work that is hard to do in isolation, but can move at lightning speed at a sprint -- where you can call on a lot of community resources. So, we all owe a big thanks to the Seattle Plone Users' Group, our organizers, and all the sprint sponsors, particularly including the Plone Foundation and Chris Calloway.

Alex Clark: Introducing Plock Pins

$
0
0
"Plock Pins are the greatest thing to happen to Plone since Buildout"— Alex Clark

Plock Pins are the final incarnation of a collection of Plone Buildouts I've been developing since 2010. They attempt to make it easy to install Plone with only Python installed [1]. Installing Plone with Plock Pins looks like this:

$ virtualenv-2.7 .
$ bin/pip install zc.buildout
$ bin/buildout init

Now edit your buildout.cfg to add a link to extend the Plock Pins:

[buildout]
extends = https://raw.github.com/plock/pins/master/plone-4-3

Then run Buildout:

$ bin/buildout

Followed by the following command to start Plone:

$ bin/plone fg

That's it! This technique should work for all versions of Plone all the way back to Plone 1.1. If it doesn't, please let me know here:

Finally, here is a list of all available Plock Pins you can extend [2]:

[1]In future versions of Python 3.x, the Pip installer will be included as part of the core software. This will lessen the burden for users attempting to install packages from PyPI. In 2.x, one must install Setuptools and/or Pip before attempting to do so.
[2]Remember to use Python 2.4 for < Plone 4

Plone Emerald Sprint: Emerald Sprint report: Users control panel and User preferences

$
0
0

I’m back from another successful annual West Coast Plone sprint: the Emerald Sprint on Whidbey Island near Seattle. Don’t miss Trish’s lovely video montage of our weekend:

 

I worked with Ross Patterson and Fulvio Casali on the users & groups control panel. The plan is to make it look and work similarly to the new folder contents UI, with the data getting loaded as JSON and manipulated by javascript:

New users control panel 

Compared to the existing control panel, the new one shows more information about each user, including join date, last login date, whether the user is allowed to log in, and whether the user has confirmed their email address. It also will make it easier to apply some actions in bulk, such as adding a many users to a group or adding a role to many users at once.

So far our implementation uses the same mockup pattern as the folder contents UI (the “structure” pattern), but with different columns and actions. There’s still quite a bit of work to be done:

  • Implementing the JSON view for fetching user and group data. Ross has started on this but it’s complicated because of fetching roles inherited from groups, etc
  • Implementing the various actions that can be performed on a user or set of selected users
  • Using a simple search instead of the query builder, which is intended for content indexed in the catalog. 
  • Integrating the pattern into Plone’s control panel.

On the final day of the sprint, I worked instead on the user preferences form. Our master plan is:

  1. Rename the “author” page to be the user’s Profile. Link to it from the user’s personal menu instead of to the preferences or user information form
  2. When a user views their own profile, it will include an Edit button which goes to the Edit Profile form, which is a combination of the current user information and user preferences forms.

New profile view 

We haven’t built most of that yet though. I worked on paving the way by removing a couple of preferences that aren’t needed any more: the external editor setting (the sitewide setting should be good enough) and the “visible ids” setting (whether a user can edit the id of an item from its edit form).

The latter isn’t necessary any more because I added a “Short name” behavior (plone.app.dexterity.behaviors.id.IShortName) to Dexterity, which adds a Short Name field to the Settings tab for any user who is allowed to rename items. If not specified, the id will be obtained automatically in the same manner as before. Hopefully this behavior can be enabled for the default content tyeps in plone.app.contenttypes soon.

Editing the id 

There’s still plenty of work to do, but I’m proud of the good planning that happened at this sprint. For some other sprint reports, see these posts:

  1. Redesigning Plone 5’s control panel - Cal Doval
  2. It takes a village - Trish Ang
  3. User Property Frolics on the Sound - Steve McMahon
And then there was a big group of people working on the login and registration process…can we have a report please?

UW Oshkosh How-To's: How to create an Intranet sandbox

$
0
0

The Quick Way

Go to http://plone3.webcluster.uwosh.edu:15082/sites4/manage_main and Copy then Paste the site called ".sandbox-template-site".

The Manual Way

Create a Plone site with

uwosh.policy.ldap
uwosh.intranet_theme
uwosh.securelogin
uwosh.secureloginportlet
and install these products
 	DataGridField 	1.8b2	1.8b2 
 	Easy Template 	0.7.10	0.7.10 
 	HTTP caching support 	1.0	1.0 
 	PFGDataGrid
 	Plone Classic Theme 	1.1.1	1.1.1 
 	PloneFormGen 	1.7.1	1.7.1 
 	PloneFormGen Save Data to Content(d2c) 	2.1b3	2.1b3 
 	Workflow Manager 	1.0a3	1.0a3 
 	Workflow Policy Support (CMFPlacefulWorkflow) 	1.5.4	1.5.4 
 	jQuery UI 	1.8.13.1	1.8.13.1 
 	uwosh.policy.ldap 	0.2	0.2 
 	uwosh.securelogin 	0.1.1	0.1.1 
 	uwosh.secureloginportlet 	0.1.2	0.1.2 
 	uwosh_intranet_theme

go into the GradStudies portal_skins/custom and copy everything there then paste into portal_skins/custom for the new sandbox site

go to GradStudies/@@manage-viewlets and use as a template for the new sandbox site

Site Setup -> Mail, set SMTP server to out.mail.uwosh.edu, from address to site owner

Site Setup -> Users and Groups -> Groups tab, assign sandbox site owners to Administrator group, assign these others to the group too:

 	  Brian Ledwell (ledwell)  	 ledwell@uwosh.edu  
 	  John Hren (hrenj)  	 hrenj@uwosh.edu  
 	  Luke Scorcio (scorcl56)  	 scorcl56@uwosh.edu  
 	  Michelle Loker (loker)  	 loker@uwosh.edu  
 	  Petersen, Dan J. (petersed)  	 petersed@uwosh.edu  
 	  T. Kim Nguyen (nguyen)  	 nguyen@uwosh.edu

set the skin layers correctly for all skins to include securelogin and secureloginportlet, and to put the uwosh.pfg.d2c layer above the PloneFormGen layer so the topmost ones look like this:

    custom
    uwosh_pfg_d2c_templates
    PloneFormGen
    CMFPlacefulWorkflow
    DataGridWidget
    uwosh_securelogin_custom_templates
    uwosh_securelogin_custom_images
    uwosh_secureloginportlet_custom_images

set up NetID in acl_users.

set login_form size of login name and password fields to 8

disable the "forgot your password?" on login form by going to ZMI Security tab, finding "Mail forgotten password" and unchecking Acquire

make visible the portal_actions/user/login

portal_view_customizations to add site title to the plone_logo; tweaked like this

<div metal:define-macro="portal_logo"
     tal:attributes="class string:cell width-1:4 position-0" id="logo"> 
<a id="portal-logo"
   title="Home"
   accesskey="1"
   tal:attributes="href view/navigation_root_url;
                   title view/navigation_root_title"
   i18n:domain="plone"
   i18n:attributes="title">
    <img src="logo.jpg" alt=""
         tal:replace="structure view/logo_tag" /><h1 id="siteName">Graduate&nbsp;Studies</h1></a>
</div>

May have to adjust ploneCustom.css for #portal-logo margin: 15px and #siteName top: 20px

enable cacheing, set cacheing proxies, set rulesets to Moderate

disable portal_css development mode!

change default front page content

add a plonedev.uwosh.edu URL

 

TO DO

add site owner to site owners list or intranet site owners list?

add to Plone site wiki in Intranets section

add Groups to drop down menu

 

Kagesenshi: Life updates

$
0
0
So .... I think by now plenty of people have known that I left Inigo Consulting a few months ago. Those who are wondering why, the reasons was primarily due to personal mental health reasons, which I identified the core source was that I pushed myself a bit too far, investing time and emotion on a business model that is not going to work out for my long term personal goals.

I realized things are not going to the right direction for about a year plus, but my philosophy of persistence kept me pushing it - possibly beyond my limits and nearing total mental breakdown. This led to generalized anxiety disorder, which destabilized myself, and led to other problems that cause plenty of things around me to collapse and destroyed, affected me emotionally (which, again, I did not realize my limits, due to I have a bit of alexithymia - inability to accurately identify complex emotions besides basic happy/sad) made me push myself into a mild form of posttraumatic stress disorder which keep haunting me even after 6 months.

Took me 3 months struggling with GAD/PTSD aftermath where I was really dysfunctional , before I finally decide to seek professional help, and now on Alprazolam / Xanax therapy to help balance out the calming chemical in my brian, and recalibrate it to be able to handle stress and anxiety again.

Decided to make a drastic changes in my life, started with getting rid the core anxiety source, so I quit Inigo, revamping a lot of things in my room, and in the process of selling my shares in Inigo to get back my previous time investments, an to fend off and get rid of any financial related commitments while I figure out whats new to do. I would probably be better off as an independent dev, however, I lack the understanding of full business process and sales cycle flow due to being shielded from clients during in Inigo, I have to start from scratch on this one and pick up the skills elsewhere first.

Been on it for 3 months now, at about 1mg-2mg dosage per day, and its been really helpful in keeping myself functional. I would probably be on this for the next few months to a year before I can slowly cut down on the dosages.

I'm currently in heavy research mode on Hadoop/BigData ecosystem for a Proof of Concept implementation for Abyres Sdn Bhd. Not sure about joining them full time yet, as I'm currently joining as an intern. Waiting for paperworks to be out first before giving my final decision. Nonetheless, I learnt quite a bit of business process, client management processes, leveraging partner skills and a version of FOSS business model, simply by observing the clients, the team. The most lesson I learnt was from Jeff of Hortonworks, where he help me to understand the separation of development, product, services and  sales and how these 4 components can be separated into several companies.   These knowledge are really useful if I were to decide to run my own company again in the future, or perhaps a return to Inigo.

I will be blogging about them more in the future, after I get this Tech Discovery POC fully running and I will share with you what I have learnt and discovered.


P/S: I particularly like the reduced fear/reduced inhibition effects xanax have. Somehow I feel like I'm back to my university days, where I'm willing to take the risk of doing something big , cool, and which others are sceptical and afraid to try. For example, dealing with large corporations for a big project.

To those still in Inigo, all the best. I might come back in future , either joining it back, or to create a separate branch of it, but things will be run totally differently than the model we used these past 5 years.

Kagesenshi: Reg for ZCA addicts - Part 1: Adapter

$
0
0
Recently I have been exploring Reg, a library inspired by zope.component written by Faassen, and I found its simplification of ZCA to be pretty interesting. Being really loving ZCA for a while now, I end up trying to figure out how the ZCA patterns fits if I were to use Reg in my future projects. So here are the list of items I discovered so far:

Adapters
This is the most common pattern I use with ZCA, primarily to simplify writing logic by standardizing the interface of objects I work with into a common set of behavior/functions:

In ZCA , adapters are done this way:

from zope import interface
from zope import component

class IDog(interface.Interface):
def bark():
pass

class Dog(object):
interface.implements(IDog)

def bark(self):
return 'Woof'

class IDuck(interface.Interface):
def quack():
pass

class Duck(object):
interface.implements(IDuck)

def quack(self):
return 'Quack'

class IAnimalSoundAdapter(interface.Interface):
def make_sound():
pass

class DogSoundAdapter(grok.Adapter):
interface.implements(IAnimalAdapter)
interface.adapts(IDog)

def __init__(self, dog):
self._dog = dog

def make_sound(self):
return self._dog.bark()

class DuckSoundAdapter(grok.Adapter):
interface.implements(IAnimalSoundAdapter)
interface.adapts(IDuck)

def __init__(self, duck):
self._duck = duck

def make_sound(self):
return self._duck.quack()

gsm = component.getGlobalSiteManager()
gsm.registerAdapter(DogSoundAdapter, (IDog,), IAnimalSoundAdapter)
gsm.registerAdapter(DuckSoundAdapter, (IDuck,), IAnimalSoundAdapter)

dog = Dog()
duck = Duck()

dog.bark() # Woof
duck.quack() # Quack

IAnimalSoundAdapter(dog).make_sound() # Woof
IAnimalSoundAdapter(duck).make_sound() # Quack

And that is quite a code, which utilizes classes, and large external interfaces libraries and component libraries which might can make many pythonistas cringe. 

In Reg, things are simplified. Similar functionality is done not through interfaces and complex object/class markers, but through simple functions. Reg adapter interfaces are simply functions:

import reg
import reg.implicit


class Dog(object):
def bark(self):
return 'Woof'

class Duck(object):
def quack(self):
return 'Quack'

@reg.generic
def make_sound(obj):
raise NotImplementedError

def dog_sound(dog):
return dog.bark()

def duck_sound(duck):
return duck.quack()

registry = reg.Registry()
reg.implicit.initialize(registry)

registry.register(make_sound, [Dog], dog_sound)
registry.register(make_sound, [Duck], duck_sound)

dog = Dog()
duck = Duck()

make_sound(dog) # Woof
make_sound(duck) # Quack

Now, ZCA users might shout by now, "I want class based adapters!" (I know I did). This is how it can be implemented in Reg:

import reg
import reg.implicit

class Dog(object):
def bark(self):
return 'Woof'

class Duck(object):
def quack(self):
return 'Quack'

@reg.generic
def animal_sound_adapter(obj):
raise NotImplementedError

class DogSoundAdapter(object):

def __init__(self, dog):
self._dog = dog

def make_sound(self):
return self._dog.bark()

class DuckSoundAdapter(object):

def __init__(self, duck):
self._duck = duck

def make_sound(self):
return self._duck.quack()

registry = reg.Registry()
reg.implicit.initialize(registry)

registry.register(animal_sound_adapter, [Dog], DogSoundAdapter)
registry.register(animal_sound_adapter, [Duck], DuckSoundAdapter)

dog = Dog()
duck = Duck()

animal_sound_adapter(dog).make_sound() # Woof
animal_sound_adapter(duck).make_sound() # Quack

Thats it for now, next one I will probably share about multiadapters on Reg.

Andreas Jung: Do you have your EuroPython 2014 ticket already?

Plone.org: A Very Brief Overview of the March Emerald Sprint

Cosent: Designing an open source social intranet

$
0
0

Netsight have invited Cosent to collaborate on designing a complete social intranet software suite, to be developed in collaboration with the open source Plone community.

A "Plone Intranet" summit in the wake of the 2013 Plone conference listed user experience, that is: design, as the single most important challenge to tackle if we want to strengthen Plone's attractiveness for the intranet market.

As everybody knows, design is not a problem one solves in a committee. We're using a hybrid model of collaboration styles that allows us to combine the design strengths of a core team with the scaling capabilities of an open source community.

The plan is to donate the work we've been doing to the Plone community, so come and join us at the Plone Open Garden 2014 in Sorrento to get involved.

So, what have we been up to?

Last winter Cosent published the Digital Workplace Technology Roadmap.

Following up on that, we've been analysing the competition, both in terms of the user experience their platforms offer but also in the kind of problems they solve, i.e. what markets they're in. We see significant market potential for a Plone-based solution.

Additionally, we've analysed dozens of cases studies of award-winning intranet designs and have clustered hundreds of intranet screenshots to understand common functional areas, or landing pages, in intranets, mapping those against the model provided by the Digital Workspace Technology Roadmap.

Last week, Netsight and Cosent have been sprinting to turn the insights gained from all of that into actionable designs, that can be used to guide software development.

We selected three types of landing pages in intranets for deeper investigation. For each of these pages, we brainstormed specific functions that users would want to use and card-sorted those into families of similar functionality.

Matt and Lewis at the whiteboard

We then picked a single landing page to work on and created several epics with short scenarios about a typical sequence of actions a user would execute to obtain a specific outcome. For example, one of our epics is:

(Team Member) Wendy receives an email from Peter with a list of questions and data that need to be collated before the next meeting of the project board. She forwards the mail into the intranet, where she flags it as a todo for next week on Project X, tags it as "board meeting", adding a note with some initial ideas and could @marcella maybe share her thoughts on this?

For each epic, we created a diagram that sequenced every function invoked as part of the scenario, and then expanded each function step into a full-fledged user story. For example, one of the steps halfway the above epic is the following user story:

Team Member can mention other Team Members in the note (using '@' syntax).

Don't shuffle the stack

Fleshing out those user stories was a lot of work, and involved detailed discussions about our assumptions and choices regarding security architecture and overall strategy. This was done by part of the team, while the other half worked on wireframing possible solutions for the epic. That was a bad idea. They had the same discussions, with different conclusions.

elements of user experience

Moving from epic to wireframing involves jumping a level up the design stack in the Garrett five-level model of the design process. When we brought the finished user stories together with the wireframe sketches we had some major inconsistencies. This appears to confirm the model and indicate that you need to get your foundations right before moving to more concrete designs. In this case, you really need to define your scope in detail, before wireframing solutions.

After re-syncing our minds and merging our work, in the final day we ventured into wireframing territory not for a whole page, but for exploring a set of micro-interactions that form the core of a cohesive social intranet experience. We also deepened our understanding of user needs and elaborated on the personas we're using to drive the design.

All in all, we feel we have not only made significant progress towards valuable design outcomes, but also have prototyped a repeatable design process that tackles very complex design challenges in a systematic way.

We plan to have another design sprint in a few weeks to prepare for Sorrento, and look forward to sharing our work there. See you in lovely Italy!

JC Brand: New spelling and grammar checker for TinyMCE

$
0
0

The spellchecker "After the deadline" is now available as an add-on for TinyMCE in Plone. It's multilingual, open-source, platform agnostic and even does grammar checking.

I've actually already added this add-on two months ago but didn't get around to mentioning it.  After the deadline is an open-source spelling and grammar checker that can be integrated with web based WYSIWYG editors, such as our beloved TinyMCE.

This new spelling checker provides some advantages over the existing (and still default) IESpell. Firstly, IESpell only works on Microsoft Windows. Secondly, IESpell is only free for non-commercial purposes. After the deadline however is open-source (GPL license, see here) and platform agnostic. Oh, and it's multilingual as well! To top it off... the killer feature, for us at least, is that it has a grammar checker. Pretty impressive IMHO.

The actual thing doing the spellchecking, is a Java server app, which you should download and install on your own server. There is a public default option for testing and demonstration purposes, but it comes with no guarantees.

After the deadline is included with all Products.TinyMCE releases since 1.2.1, but it's not the default spell checker. To enable it is at least straightforward.

Enabling AtD in TinyMCE for Plone:

  • Go to the Plone control panel and click on "TinyMCE Visual Editor"
  • Click on 'Toolbar' (middle left)
  • Make sure that 'spellchecker' is checked.
  • Click on 'Libraries' (top right)
  • Under Spellchecker plugin to use, choose After the deadline
  • Under AtD Service URL, choose your ATD server's URL. (The default is their - public service)
  • It's however recommended that you install your own AtD spellchecker service
  • See here for more details.
  • You should now have After the deadline enabled and have a spellcheck button in your TinyMCE editor.

If you'd like a demonstration of ATD, click here.

Oh, and if you find any spelling/grammar mistakes in this blogpost, it's because I'm still using Products.TinyMCE 1.1.6 ;)

UW Oshkosh: 5 days to go! talk proposals & early bird registration for Plone Symposium Midwest

$
0
0

Don't be fooled!  There are only 5 days to go to our April 1 deadline to:

 

Don't miss:

 

 

"Plone 5: Ship It - Ship It Good!" will be the theme for our sprints :)

 

 

Come learn, share, and make great connections!

 

    The Plone Rangers at UW Oshkosh

 

Mikko Ohtamaa: Linux server ghetto duplication

$
0
0

This blog post is about how to duplicate your Linux services and configuration from one server to another. We use simple and hackable SSH, rsync and shell scripting to copy the necessary files to make a cold spare from your existing server installation.

Screen Shot 2014-03-26 at 00.17.09

1. Preface

The method describes here is quite crude and most suitable for making a spare installation of your existing server. In the case you lose your production server, you can boot your cold spare, point your (fail-over) IP address to the spare server and keep business going – something you might want to do if you run mom’n'pop web hosting business. Because of the simplicity of the method it works on default Linux installations, bare metal servers and old-fashioned file systems like Ext4.

The instructions here have been tested with Ubuntu Linux 12.04, but should work with other versions with minor modifications. I have used this method successfully  with Heztner hosting (highly recommended for all the cheapskates out there) by having one production machine and one spare machine. The spare is mirrored weekly. Daily Duplicity backups can be restored on the spare if week is too long data loss period. Though in my case the servers are bare metal, the method works for VPSes as well.

The duplication script might be also useful for setting up a staging server from your production server for software developer and testing.

2. More powerful duplication tools

More fail safe, more engineer-oriented duplication approaches exist, but usually require additional preparation and tuning on the top of the default Linux installation. Applying these to existing running Linux installation might be tricky.

3. Building and running the duplication script

This script is run on the target server (cold spare) and it will clone the content of the source server (actual production machine) on itself. It uses SSH keys and SSH agent to create the initial connection, so make sure you are familiar with them.

Assumptions

  • The target server must be clean Linux installation, the same exact version as on your source server.
  • Your server has standard /etc/passwd user account management. This is copied first so that we correctly preserve file ownership (related ServerFault discussion).
  • Services you run (PHP, Django, Plone, Node.js, you-name-it) are under /srvas recommended by Linux filesystem hierarchy standard
  • Source and target MySQL servers must have initialy same password for the root. Set this when the script runs apt-get install mysql-server for the first time.

Limitations

  • The first run is interactive, as apt-get install asks bunch of stuff for packages like MySQL and Postfix.
  • MySQL dumping and loading the dump does not guarantee all of your data survives intact, especially when skip-lock option is used for the performance reason. For ordinary CMS operations this isn’t a big issue.
  • If you other services beside MySQL needing special flush or lock handling follow the related instructions.

For MySQL duplication make sure you have the following /root/.my.cnf file on the source server, as it allows you to interact with MySQL:

[mysqldump]
user=root
password=YOUR PASSWORD HERE

[client]
user=root
password=YOUR PASSWORD HERE

Run the script inside a screen, because the duplication process may take a while and you do not want to risk losing the process because your local internet connectivity issue.

scp mirror-ubuntu.bash targetserver:/tmp
ssh targetserver
screen 
cd /tmp
bash mirror-ubuntu.bash

4. Testing the spare server

After the duplication script has successfully finished mirroring the server you want to check if the services can be successful started and interacted on the cold spare.

Change internet-facing IP addresses in the related services to be the public IP address of the spare server. E.g. the following files might need changes:

  • /etc/default/varnish
  • /etc/apache2/ports.conf
  • /etc/apache2/sites-available/*

Spoof your local DNS to point the spare server on the tested sites. Edit your local /etc/hosts and add spoofed IP addresses like:

1.2.3.4 www.service1.example.com www.service2.example.com opensourcehacker.com

Access the sites from your web browser, see that database interaction works (login) and file interaction works (upload and download files).

5. mirror-ubuntu.bash

#!/bin/bash
#
# Linux server ghetto duplication script
# Copyright 2014 Mikko Ohtamaa http://opensourcehacker.com
# Licensed under MIT
#

# Everything is copied from this server
SOURCE=root@myserv.example.com

# Our network-traffic and speed optimized rsync command
RSYNC="rsync -a --inplace --delete --compress-level=9"

# Which marker string we use to detect custom init.d scripts
INIT_SCRIPT_MARKER="### BEGIN INIT INFO"

# As we will run in screen we need to detach
# from the forwarded SSH agent session and we use a local
# SSH key to perform the operations.
# Also overriding /root destroys our key.
# Create a key we use for the remaining operations.
TEMP_SSH_KEY=/tmp/mirror_rsa

# The software stack might have touched the following places.
# This list is compliled through trial-and-error,
# sweat and cursing.
# We cannot take /etc as a whole, because it contains
# some computer specific stuff (hard disk ids, etc.)
# and copying it as a whole leads to unbootable system.
CHERRYPICKED_ETC_TARGETS=(\
    "/etc/default" \
    "/etc/varnish" \
    "/etc/apache2" \
    "/etc/ssl" \
    "/etc/nginx" \
    "/etc/postfix" \
    "/etc/php5" \
    "/etc/cron.d" \
    "/etc/cron.daily" \
    "/etc/cron.monthly" \
    "/etc/cron.weekly" \
    "/etc/init.d")

# Create a key without a passphrase
# and put the public key on the source server
rm $TEMP_SSH_KEY 2>&1 > /dev/null
ssh-keygen -N '' -f $TEMP_SSH_KEY
ssh-copy-id -i $TEMP_SSH_KEY $SOURCE
# Detach from the currently used SSH agent
# by starting a session specific to this shell
eval `ssh-agent`
ssh-add $TEMP_SSH_KEY

# Assume the system have same Ubuntu base installation and no extra repositories configured.
# Bring target system up to date.
apt-get update -y
apt-get upgrade -y

# TODO: check that the kernel uname is same
# on the source and the target systems

# This is somewhat crude method to try to install all the packages on the source server.
# If the package is missing or replaced this command will happily churn over it
# (apt-get may fail). This MAY cause user interaction with packages
# like Postfix and MySQL which prompt for initial password. Not sure
# what would be the best way to handle this?
ssh $SOURCE dpkg --get-selections|grep --invert-match deinstall|cut -f 1|while read pkg
do
    apt-get install -y $pkg
done

# As some packages might have changed due to version upgrades and
# deb renames the following step needs interaction
ssh $SOURCE dpkg --get-selections|grep --invert-match deinstall|cut -f 1

# Copy user credentials first to make sure we
# get the user permissions and ownership correctly.
# http://serverfault.com/a/583336/74975
echo "Copying users"
$RSYNC $SOURCE:/etc/passwd /etc
$RSYNC $SOURCE:/etc/shadow /etc
$RSYNC $SOURCE:/etc/group /etc
$RSYNC $SOURCE:/etc/gshadow /etc

# Copy home so we have user home folders available
# Skip duplicity backup signatures
echo "Copying root"
$RSYNC $SOURCE:/root / --exclude=/root/.cache
# lost+found content is generated by fsck, uninteresting
echo "Copying home"
$RSYNC $SOURCE:/home / --exclude=/home/lost+found

echo "Copying /etc targets"
for i in "${CHERRYPICKED_ETC_TARGETS[@]}"
do
   $RSYNC $SOURCE:$i /etc
done

# Most of your service specific stuff should come here
echo "Copying /srv"
$RSYNC $SOURCE:/srv /

# Make sure stuff which went to /etc/init.d gets correctly reflecte across runlevels,
# as our /srv stuff has placed its own init scripts
for i in  /etc/init.d/*
do
    service=`basename $i`
    # Separate from upstart etc. jobs
    if grep --quiet "$INIT_SCRIPT_MARKER" $i ; then
        update-rc.d $service defaults
    fi
done

# Copy MySQL databases.
# Assume source and target root can connect to MySQL without a password.
# You need to set up /root/.my.cnf file on the source server first
# for the passwordless action.
# http://stackoverflow.com/a/9293090/315168
echo "Copying MySQL databases"
ssh -C $SOURCE mysqldump \
    --skip-lock-tables \
    --add-drop-table \
    --add-drop-database \
    --compact \
    --all-databases \
    > /root/all-mysql.sql

# MySQL dump restore woes
# http://stackoverflow.com/a/21087946/315168
mysql -u root -e "SET FOREIGN_KEY_CHECKS = 0; source /root/all-mysql.sql ; SET FOREIGN_KEY_CHECKS = 1;"

# Remove the key we used for the duplication
rm $TEMP_SSH_KEY

Buy open source friendly bitcoins Subscribe to this blog in a readerFollow me on TwitterFollow me on FacebookFollow me Google+

Plone Emerald Sprint: plone.login: updating Plone’s login

$
0
0

I worked with Franco Pellegrini, Chris Calloway, Cris Ewing, Eric Steele, and Liz Leddy to move Plone’s login to a more modern z3c.form based implementation. The first part of this process was to discover and document the parts that needed to be updated. From this, we decided to create a html based prototype before actually getting down to coding. This planning allowed us to determine what the major tasks were, and what could be done later. 

Login:

 login-mockup.jpg

Note that we’ve included external auth sources right on the login view. Additionally, links for help and new account sign up are easy to find. External auth is not yet implemented. The login implementation is working, but may not handle all the cases that the current login functionality has.

 

Signup/Registration:

 signup-mockup.jpg

Two things to note is the ability to use an external source for your registration and the gray bars at the bottom is the password strength indicator. Both of these are not yet implemented.  

Login Help:

trouble-mockup.jpg 

 The login help form has been stubbed out but needs a full implementation.

Logout, Insufficient Privileges, Request Access have all been implemented, but require further attention. Desired improvements to the login system include a way for integrators to easily include their own organization’s custom password policy.

 

To see it in action follow the instructions to get the Plone 5 cordev buildout working, then run  

./bin/buildout -c  plips/plipNNNN-new-login-flow.cfg

 

 

 

 

 

 

 

 


Plone Emerald Sprint: User Data Schemas

$
0
0

It’s been a few weeks since the conclusion of the Emerald Sprint.  Steve has already provided an excellent account of the work we did and I just want to add a little to that. 

First, I have to say sprinting is awesome. So much goes on at a sprint beyond straight up coding: friendship, mentorship, and knowledge percolate and take hold in no other fashion I’ve experienced before. If you have an opportunity to join a sprint, do so as you won’t regret it!

Working with Ian Anderson and Steve McMahon was great. We were able to take and re-factor the existing work done to make user data schema editable through the web, and allow it to be updated via Generic Setup with package installation. But there is still some work to be done: 

  • Currently, image and file fields do not properly store or reference their file objects. 
  • We need the ability to break the required fullname into a computed field based on firstname, middlenames and lastname fields (which also need to be reorderable due to regional differences).

Before signing off, I’d like to take a moment to thank Fulvio for bearing the brunt of organizing and managing the sprint. Sally also deserves a round of applause for taking charge of the dining logistics and to those that provided their expertise in the kitchen (Sally, Ross, Spanky, Liz, Fulvio, Cris, and Franco) to sustain the sprinters. Just ask any of the sprinters about Franco’s Tuco sauce, Sally’s oyster pasta, Spanky’s salsa and guacamole, or Ross’ fabled Aviation.  There really needs to be a Sprinter’s Cookbook for future endeavors.

Plone.org: Plone Symposium Midwest 2014 Needs You

Wichert Akkerman: Task queues

$
0
0

When writing a web application or REST backend I occasionally run into a common problem: sometimes I need to do things that will take a while: sending a push notification to mobile apps, generating emails or doing an expensive calculation. Doing this immediately would result in very long response times, which is not acceptable. That means I need a way to offload those to something

Read entire article.

Plone.org: The Push for Plone 5: 8 Sprints by July

$
0
0
Summaries and (lots of) links to this year's past and upcoming sprints.

Reinout van Rees: Print(reinout) for debugging

$
0
0

Quite some colleagues use my name to raise an error. It is a trick I learned them :-)

There are a large number of debugging tips and tricks. import pdb; print statements; a full-blown IDE with an interactive debugger; logging; etc.

Often the simplest and most useful thing to check is to first verify that you're looking in the right place at all. Why? Well, you might be working on the right python file, but are you really sure that your web app isn't using a previously installed package instead of your checked out developer version, for instance?

Or you work on some method, but another class overrides the method you're debugging.

In cases like this I often add a print(reinout) statement somewhere in the function. That's not a typo. It definitively should not be print("reinout"). The latter only prints reinout, the first prints:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'reinout' is not defined

Much easier to spot. Makes it obvious that at least you're looking in the right spot!

Viewing all 3535 articles
Browse latest View live