Drupalcon 2010 San Francisco Notes
Here are my notes from the conference. See:http://sf2010.drupal.org/conference/schedule for all available sessions and videos.
The Bird's Nest: Behind the Scenes of Twitter
Twitter receives over 50 million tweets per day, with over 107 million registered users. Scaling to that size is possible through one half architecture and one half culture. This session will take a brief stroll through Twitter's system design, then walk through the engineering practices that makes it possible. Topics include: agility, parallel development, testing, large scale deployment, and the overlooked human elements
No Video
Notes
- Large scale, hundreds of servers
- Can manage 4+ deployments a day
- Site Features rollouts: using bitorrent (peer to peer file transfer software) to update servers
- Team morale is a factor: positive, engaged, passionate, balanced
- Contributions to open source community: flock, gizzard, kestrel
- AGILE Tools: pivotal tracker
- CDN (Content Distribution Network, aka Akamaii): Speedboost, also multiple virtual hostnames
- Moved to Amazon Cloudfront from Akamaii
- http://twitter.com/jobs - job hiring approach
- http://twitter.com/about/opensource - contributions
Make Drupal Run Fast - increase page load speed
Has video
What does it mean when someone says "My Site is slow now"? What is page speed? How do you measure it? How can you make it faster? We'll try to answer these questions, provide you with a set of tools to use and explain how this relates to your server load.
We will cover:
- What is page load speed?
- Tools used to measure performance of your pages and site
- Six Key Improvements to make Drupal "run fast"
++ Performance Module settings and how they work
++ Caching - biggest gainer and how to implement Boost
++ Other quick hits: off loading search, tweaking settings & why running crons is important
++ Ask your host about APC and how to make sure its set up correctly
++ Dare we look at the database? Easy changes that will help a lot!
- Monitoring Best practices - what to set up to make sure you know what is going on with your server
- What if you get slashdoted? Recommendation on how to quickly take cover from a rhino.
Notes
- Allen Freeman and the presenter recommend looking into Mercury (http://www.getpantheon.com/mercury) or Acquia (http://www.acquia.com, Dries' company) for enhanced Drupal hosting
- Google includes page speed in relevancy / ranking
- Check out:
- Yahoo's Yslow (for firebug),
- Google's Page Speed – focuses on AJAX and more,
- www.WebPageTest.org,
- Apache Bench
- Jmeter
- Devel module in Drupal
- Drupal Parallels module - Allows for parallel downloading of the various resources inside your html document. (http://drupal.org/project/parallel) Can also look into CDN Integration module. (http://drupal.org/project/cdn)
- SimpleCDN
- www.BrowserScope.com - chart number of downloadable objects
- Dsquid / Varnish, Mercury project
- Cache Router – choose which cache system to use
- Boost – caches in files versus in database tables. (sounds like pubsys)
- Back end caching: APC ( Alt PHP Cache), also see, Xcache & EAccelerator (http://eaccelerator.net)
- Make performance improvements iteratively: Make 1 small change, test performance
- Drupal 7 will be moving to InnoDB tables
- Search: Aquia SOLR, Google Custom
- Sessions: Set to timeout 0 (session.gc_maxlifetime and session.cache_expire)
- Look into setting watchdog to write to native file system instead of to DB
- Monitor trends: Nagios, other 3rd party tools
- Use Google Webmaster Tools to show % rank for performance
- Presenter's blog: www.linuxadminblog.com
- Site: www.isupportdigital.com
- Host: www.promethost.com
Drupal security for site administrators and beginners
Has video
Drupal is a powerful tool that can be configured many ways, sometimes insecurely. Knowing popular attack vectors against your site and how to protect against them is important for everyone building a Drupal site. In this session I'll discuss some ways sites get attacked and what you can do to maintain your site's security through smart configuration and how to recover should it be attacked.
About the speaker:
Ben Jeavons has been working and developing with Drupal for over three years. Ben is a member of the Drupal Security Team, the author of the Security Review module (http://drupal.org/project/security_review) and blogs about security on http://crackingdrupal.com.
If you are interested in writing secure code and more advanced security topics be sure to check out http://sf2010.drupal.org/conference/sessions/drupal-site-security-coders-and-themers)
Notes
- XSS – Cross Site Scripting attack example shown. Runs when admin / user #1 views a malicious user comment on an improperly configured site. Site was taken offline and admin's email address and password were changed
- Some attack types: Access bypass, Cross Site Source Forgery, SQL Injection
- Prioritize security review: Config, Custom Code, Community modules
- Ongoing security review: Stay ahead of the pack, keep software up to date
- Provide only necessary permissions to users. Set to least privileges possible.
- This is why only the Admins/User#1 should be allowed to modify input filters. So no attacks can grant other users this privilege.
- "Update Status" module can inform you of security updates. This is part of Drupal 6 core
- Security advisories: Get on the email list, follow the RSS feeds and the Twitter stream
- FTP: avoid this as plaint text is insecure. Use SFTP instead.
- Check Drupal Security Advisories: http://drupal.org/security
- Check Drupal's Security Review module: automates checking many of the configuration errors that lead to an insecure Drupal site. http://drupal.org/project/security_review (Not for Drupal 5)
- Secure file system: settings should not be seen other accounts on the server
- Be careful when displaying error messages
- If SQL error message displays on page /w SQL command syntax, hackers may be able to use this to do a SQL injection attack. For example, if the error is happening because of a poorly written module passing user provided text as a request to the DB, the error message may show enough to determine how to modify the input so it passes a valid request to the DB for secure information.
- DB prefixes may help obfuscate table names
Drupal site security for coders and themers
When you put your Drupal site online you expose it to a wide world of hackers who have the potential to attack your site. In this session you can learn about some of the most common problems with code in a
Drupal site so you'll recognize them and be able to protect against them. Join Drupal security team members Greg Knaddison and Peter Wolanin as they give you the tools to hack your site and the knowledge to protect it.
You will specifically learn how to exploit a Cross Site Scripting vulnerability and Cross Site Request Forgery vulnerabilities and then also learn how to protect your code/sites against them.
This shortened free version of the full day pre-conference security session. http://sf2010.drupal.org/node/5213
Notes
- We should probably buy the Cracking Drupal book
- Check Drupal's "forward to your friend" functionality
- www.Cenzic.com - security site
- See: www.drupalceuritysupport.com
- XSS Example: Hackers can manipulate JS in unexpected ways. By adding JavaScript to the Firefox browser ID using a plug-in, this info can be stored in Drupal's browser info area. When an admin reviews the list of most popular browsers, the JS executes.
- Before displaying something, always ask: Where does this text come from? Can a user change this? In what context is it used?
- No Image tags in user content: Showed an example of why allowing the <IMG> tag in user content is bad. The browser will try to call whatever is in the image's SRC, not just image files. If a certain user module tool is installed, it creates a page with a list of users and a delete link next to them. If a user changes the image's source to the URL to delete a user by their UID, this would happen as soon as an admin tries to see the malicious comment.
- Tokens: Explained how tokens are used to authenticate legitimate form submissions. If creating a user accessible form, make sure to include the token functionality via drupal_get_token() or use the form API as it is auto protected. This way, Drupal will only accept the submitted form's contents if the form transmits the expected token.
- See: www.drupal.org/security-team/risk-levels, www.drupal.org/security/contrib
- Arbitrary Code Execution: Can sometimes slip through via problems when using pregex syntax
- SQL Injection: Make sure to use the Drupal API calls when handling SQL
- Make sure to use db_rewrite_sql() so that variables passed must be cleaned
- DO NOT construct SQL strings w/ concatenated variables. Use % modifiers: http://api.drupal.org/api/function/db_query/5
Drupal Publisher's Panel
Has video
Online News is a natural fit for Drupal. Publications of all types from newspapers, magazines, trade publications and journals have used Drupal aggressively to better showcase breaking news, analysis, features, blogs and user generated content. These publishers become avid fans and some of the best evangelists of Drupal because it answers so many of their growing needs.
Demands on all news and information organizations are growing. Despite dwindling numbers of traditional media outlets, competition for readers is greater than ever. Busy editors are being asked to do more with less resources, and a publisher’s ability to respond to trends in online news and information delivery is critical to success with readers.
This session will feature a panel discussion with Drupal implementers at prominent news sites, including The Slate Group, The New Republic and The Nation, who will address:
· Why they chose Drupal
· What aspects of the platform are most useful to them
· What tricks and tips they have for others implementing Drupal for news publishing
· Where they hope the Drupal platform can better serve the needs of publishers
This session is for anyone who is using Drupal for publishing or preparing to implement a publisher's site. We will address some architecture and module level topics, but the session is appropriate for all technical levels.
Notes
- The Economist, The Nation, The Osder Group, Phase2 Technology
- SOLR is gaining importance
- Some complaints about Drupal's flexibility versus continuity and need for code level changes to implement seemingly simple things.
- "Disposable" Content: How can we add a system for handling markup that doesn't need to be in the database, for example by an advertiser?
- Compared to previous systems:
- Community of contributors is a benefit
- Flexibility is both a benefit and a challenge
- Monetization
- Try to supply solutions that Ad agencies can't
- Try to focus on
- Optimization,
- Navigation tweaks,
- Guiding audience,
- Understanding audience
- Brand contributions to premium CPM
- Economist is trying 3 months free or current week pay or metering. Site is best source of new print subscriptions.
- Dream Tools?
- Audience management:
- acquisition through: Kindle, mobile web / iPad -> Dashboard(?)
- Need to have ability to track plan and implementation
- Balance between forward looking tech adopters that are also traditionalists
- Easier to integrate data feeds (i.e. Olympics coverage preparation)
- Stakeholder management module: A magic guide to force the requirements to match Drupal's abilities
- Audience management:
- New Tech Trends
- App versus site: Compare a set of bookmarked site pages to an app to see how stakeholders perceive the difference (if any)
- Really need to ask audience for input before committing to massive undertaking
- Modules: Managing Expectations regarding availability of modules
- Need to be very selective about which modules get installed due to performance penalties. I.e. Compare facebook Connect impact on usage versus performance
- Marketing Tools in Drupal
- Take advantage of feeds: how does that work with Facebook Connect where new logins are not the same type of customer
- Scalability / Dealing with heavy traffic bursts
- Economist: 25 million monthly / 4 million uniques
- Pressfow: performance optimized Drupal install. Look for other speed optimized versions
- Memcache,
- Varnish reverse proxy cache (http://varnish-cache.org/)
- ESI's to help w/ authorized users load is coming in Drupal 7. This may be available for Drupal 6(?). Edge Side Includes are a markup language used in frontend webservers to define which components of a webpage may be cached. (http://en.wikipedia.org/wiki/Edge_Side_Includes)
- Staging Server Issues:
- Check OpenPublisher for workflow
- Files / Assets: Still a challenge and integrating Cloud Hosting
- Integration with print: Not really happening yet
- Economist is using CCI
- Ad Management: DFP (Dart for publishers) and Google Ad Manager
Defining and pitching your process
Has video
Effectively communicating how you work is a powerful way to convince clients to choose you. Before they make a decision, potential clients want to know what working with you will be like, and need to be confident that it will not only be effective, but that they'll be comfortable in the process. This session will offer insights into thinking through your own work process, how to communicate that to decision makers, and the key role this will play in setting client expectations for the project.
Notes
- Iterative Client Focused Design using an approach of:
- Discovery, Design, Implementation
- Implementation can overlap with Design
- Suggest starting w/ a Garland theme to prototype higher end functionality
- Early Prototyping using Garland requires stringent expectations setting
- Identify the areas in which you work
- Put the steps of your process in order
- Ask: Is this "really" the way you work?
- Biggest issue is usually in the order of the steps
- Simplify
- Well defined process should describe the way you work and show value to the client
- Target the work you want to do
- Focus on what's best for the client
- Simplify and improve value
- Suggest working in a single design direction rather than multiple design comps. This helps avoid the phenomenon where clients pick pieces of each design and try to combine them into a new request.
- Manage expectations
- Find the rough spots in the process. Could problems have been avoided?
- Pursue great projects and clients. If it is a bad fit, walk away.
- Communicate your process: converting actual value in the process to perceived value by the client. It's critical to explain the importance of wire framing.
- Understanding the way you work distinguishes your proposal in the mind of your client. Even if it very tough to get through to them, calmly show and explain the process, cite examples, etc.
Build a Powerful Site Search with the User-Friendly, Easy-to-Install Search Lucene API Module Suite
Has video
Abstract:
The Search Lucene API module provides a fully integrated, easy-to-install alternative to the core Search module. Because it makes use of the Zend Framework’s PHP port of the Lucene search engine library, installation takes minutes and requires no external services making it a cost-effective advanced search solution. In addition, the module has faceted search capability which allows users to filter results via an intuitive interface. Contributed modules such as Search Lucene DidYouMean and Search Lucene MoreLikeThis correct spelling errors and recommend content based on the words in the index respectively. As a result, visitors are directed to relevant content beyond what their initial search query provided.
Audience:
Attendees must know how to install and configure Drupal modules, but programming knowledge is not required to view this presentation.
Goals:
After viewing the presentation, attendees will be able to install and configure Search Lucene API along with the most popular contributed modules to build a powerful, user-friendly site search.
Topics:
· Introducing the technology and capabilities of Lucene at a high level
· Installing and configuring Search Lucene API to index and search your site’s content
· Configuring faceted search using the core Search Lucene Facets module
· Adding spelling suggestions based on the terms in the index using the Search Lucene DidYou Mean module
· Displaying content recommendations based on the text and metadata of the node being viewed using the Search Lucene MoreLikeThis module
· Localizing the site search using the Search Lucene Internationalization project
· Indexing CCK fields and exposing them as facets using the Search Lucene CCK and Search Lucene Facet Builder modules
Notes
· Can handle wildcards, fuzzy, proximity searches, content recommendations
· Hosts index on file system to remove DB involvement and prevent search from degrading site performance
· Uses PHP port of Lucene, not need to deal w/ JAVA
· Targets sites of up to 5,000 nodes (by node size, not true node count, based on experience of presenter)
· 3.0 API will integrate w/ JAVA
Installation
· Seems to support Drupal 5.x
· Requires 3rd party library from sourceforge.
· see http://zugec.com
· Docs in Drupal handbook
· try search lucene content and search lucene api modules
· Setup
o Logging level - should be on during devel only
o permissions may need to be flexible based on php.module status
o allows configurable default operator
o Needs periodic optimization. new content creates a new file in index. Optimize converts them all to one file
o Result Set limit can help increase max nodes by stopping the results returned after a certain amount
o can configure ranking by page area where text is found in (title, body, etc)
o Results sorting modifiable by hooks. So probably programmatic
o *** Faceted search example in slide shows search w/ counter by results in rail. i.e. by content type. Requires an additional module. See http://cmsreport.com/search/luceneapi_node/drupal
o Can modify to ignore a result if the search term is not indexed x number of times
· Porterstemmer: can be setup w/ this. DidYouMean can sometimes recommend word stems (i.e. Body => bodi or something like that)
· Search Lucene CCK - currently in Alpha 3
· Search Lucene Facet Builder - under construction but will eventually allow faceting CCK content
· Search Lucene Views - being rebuilt
· Does not search attachments w/o a 3rd party parser to get the text out. After that, there are hooks that developers can use to index.
· Possible to index non-node objects through dev efforts to extract text. Indexing Views is tougher because it's data that's changing.
· Module Incompatibility:
o Can't be combined w/ other search solutions right now. Difficult to transition from one to another (i.e. SOLR)
o Drunken Monkey is doing a Google summer of code project to do a generic search API
· Apache SOLR vs Lucene:
o Number of nodes
o Client desires enterprise level solution
o Both have very good support. Lucene is built on Zend framework
Developing Apps for iPhone / iPad / Android using drupal as Base System
Has video
With launch of iPad and NexusOne - iPhone/Android application market can't be ignored. Both these technologies has great potential to use our existing web based tools such as drupal. You can totally use drupal to manage/update your iPhone/Android native application.
Some iPhone/drupal stuff I wrote in past at: http://civicactions.com/blog/sumit
This session is about building native iPhone/iPad/Android applications using opensource web based technologies (http://www.appcelerator.com) with drupal as base system to manage: content, users, views, permissions etc.
What will be covered
- An introduction to Titanium Framework to develop iPhone/Android apps using web technologies like HTML5 and jQuery etc
- Introduction to ServicesAPI and web standards [json,xmlrpc]
- How to manage your application's data (content, tables, views, users, menus, taxonomy etc) using drupal.
- We will build a live app ... introduce you to some code here (ServicesAPI, YQL, jQuery and Titanium API)
- Demo of some apps we build for Android and iPhone platforms
- Questions!
Quick demo of what we are going to discuss: http://www.youtube.com/watch?v=GOzwOnEDinM
Notes
Developing native apps w/ Drupal as a base system using HTML, JS apps
· 40% of all internet traffic will be by mobile in 2013
· Example Demo they showed
o Something that used Drupal to host some video nodes and then the phone pulls them through a View and can play
o Simple RSS loader w/ OS app standard scrolling
o Photo viewer: load the photos via a view. Called securely using services.api and showed them in iphone coverflow view
o Crime view: shows a map on the pad pulling the data from a site. This could be an app for us to show restaurant locations?
o Modifying nodes / creating content: many types of Drupal functions are available
o Most of these demos come with Titanium and have had their URL's changed to Drupal
Focus on: (due to popularity and community support)
· PhoneGap
o WebKit
o Open Source
o Limited Access to Native functionality
o Slower Performance
· Titanium Mobile - Focus of presentation
o API is about 1.5 MB in size
o Good community support
o Open Source - Apache 2 license
o Build Native apps 9.x+
o Very Fast in performance
o Modular approach to add functionality (open source devs can add to project)
o Paid support / partner etc
o Lots of APIs available
o Free (as in beer)
o Development time of 2-3 weeks w/ 1-2 developers
Why use Drupal to Manage your Mobile App Data
· Most reliable / popular CMS
· Manages all your
o data,
o meta data and users,
o search needs and geo data
What you'll be coding
· JS (primary portion of app should be in this for speed)
o Example for creating a simple Table View is declaring some JS objects and the adding the table onto the window
· Titanium API
· WebViews: HTML 5 / CSS 3 (these are a little slow)
Incorporate Services API
· Methods inteded for consumption by remote apps
o user.get, user.save, etc
· Servers are ways a service can be called from the remote site
o i.e. xmlrpc, json, jsonrpc, etc
· Authentication methods
o Keys, and something else (open???)
How to get started
· Install iphone SDK
· Install Titanium Developer
· Create account on Tiatanium Dev
· Download and install Services and json_server modules on drupal site
Titanium API's
· accelerometer
· app
· contacts
· database
· Fb
· geoloc
· gesture
· map
· xml
· media - camera, videa, etc
· network - communicate via XHR methods
· platform - per OS differences
· ui - table views, sliders, text fields, etc. from normal mobile aps
· ui.iphone
· ui.ipad
· Utils
· Yahoo (YUI) - exposes ~300 yahoo API's
· JSON - this is in the core now
iPhone, Drupal, and Web Services
Has video
Mobile Strategy is becoming critical to websites and companies who want to provide their users with more accessible and interactive information while on the go. Iphone Apps provide a great way for these organizations to adapt their content and features to an integrated mobile experience.
With the rising popularity of these apps, we need a standard Drupal iPhone application or Library to begin working with when developing apps that reference Drupal content. Today's solution is a Library that leverages the power of Cocoa/obj-c and connects it to Drupal.
In this presentation I will demonstrate how easy it is to integrate your Drupal powered website with Services, a New Web Services module, and the iPhone SDK. I’ll go into depth about how to configure your Drupal Site in order to obtain the information you'll need for your iPhone app. On the iPhone SDK side, I will explain the pros and cons of the Library and any extensions you'll need to add for custom functionality as well as a demo of it working.
The best part is, it all uses Apples Property List format, which means you can directly make objects from a Web Services call and not have to parse any XML or JSON. This greatly reduces the battery consumption and processing power required by your app.
Notes
Did not attend this but may be useful to watch video.
Top 100 Useful Contributed Modules
Has video
Click here to download the list.
Have you ever said to yourself, "There must be a module that does this."?
This session will take a look at 100 contributed modules. With over 4,000 contributed modules available, this session will look at the top 100 most useful modules based on my experience on building Drupal websites. After attending this session, you will leave with a great list of modules that cover a wide range of functionality.
Notes
Skipping some of those we already use, CCK, Panels, etc
· Calendar / Date API
Views
· Views Gallery
· Views Rotator - god for swapping images
· Views Slideshow
· Views Carousel
· Views Tabs
· Views Bulk Operations
· Event
· Event Repeat
· Scheduler: publish nodes on specified dates
· Web Forms: create form nodes
· Add to Any: Seems to be a better share tool.
Theming
· Quick Tabs.
· Vertical Tabs: converts fieldsets into collapsed tabs in a rail. Try in admin area?
· Block Theme
· Block Class: assists in adding a class to a block w/o using block numbers
· Collapseiblock: easy collapsing block function
Images / Galleries
· ImageAPI / imagecache
· image Fupload - upload images in a zip file
· lightbox/thickbox
· Gallery
· Slideshow
· SWF Tools
Utilities
· Token: beyond security, can also be used to replace a text variable
· pathauto
· path redirect
· flag
· rules
· string overrides
· menu breadcrumbs
Chaos Tools (ctools) - Page Manager
· Required by panels
· Ajax Responder
· Modal Dialogs
· Form Wizard
· Exportable
· Content Plug-ins
Development
· Devel
· Drush – command line tool
· Coder - should look into this more
· Deadwood - can implement changes similar to those recommended by Coder
· Schema - for building table schemas, especially for install files
· Backup & Migrate
· Drupal for Firebug - adds drupal info to firebug output
· Masquerade - lets you login as another user
Administration
· Admin - D7 usability module for D6
· Admin Menu
· Advanced Help
· Features - helps a little w/ staging / production work flow by packaging changes into a module for export. Create panels, views, etc.
· Total Control Admin Dashboard
Search Engine Optimization
· Nodewords
· Page Title
· Site Map - this is for displaying a sitemap for users. Dynamic, uses menu / taxonomy
· Xmlsitemap
· Google Analytics - module for this?
· Tagadelic
Access Control
· Access Control List (ACL) - Lets you limit a node to a user?
· Content Access
· LDAP Integration
· Content Profiles - replaces regular Drupal profile as a node to add CCK fields
· Advanced Profile Kit - panels for user info
· User Points
E-commerce - Ubercart is best solution
· Ubercart discount coupons
· Ubercart Fee
· Ubercart Restrictions - limit certain products to certain roles
· Secure pages - makes sure that you are at httpS
Spam: mollum, CAPTCHA
Organic Groups: OG Block visibility, OG Menu
· Not really needed if using panels
Miscellaneous
· IE Unlimited CSS Loader - removes limit in IE on max # of styles
· Drupal for Facebook - create canvas pages that are output from facebook
· Fivestar
· Subscriptions
· Simplenews
· Mimi mail - help control emails coming from hosting server
Providing a Scalable Infrastructure for Whitehouse.gov
Has video
In October 2009, the official site of the President, Whitehouse.gov re-launched on Drupal. The preparation, development, and testing of the infrastructure required was an immense effort full of challenges. It was a landmark win for Drupal and did a lot to legitimize Drupal's flexibility, security and scalability to those outside of our community. This session will provide insight into the infrastructure required to power this uniquely visible and high traffic site.
While many of the details of this particular implementation cannot be revealed, the session will address how a site with this caliber of usage deals with traffic spikes, code and infrastructure deployments and scalability. The discussion will include strategies and techniques that include:
· Deploying Drupal in a Scalable Way
· Set up and management of deployment architecture
· Implementing replication architecture
· Building for high availability
· Preparing for disaster recovery
Notes
This presenter went through slides very quickly so notes are missing some things.
Scaling applies to many aspects: infrastructure, planning, hardware, etc.
· Wanted a scalable platform, detailed control of message presentation
· Robust core
·
· Multi team:
o Drupal Development: content conversion
o building infrastructure: load testing, performance
· Collaborating w/ over 10 agencies and vendors
· Ingredients
o Great design
o Performance patches
o D6
o Lots of contrib modules
o Custom features
· Micro-site example
o using organic groups
· Apache SOLR search
o Custom media browser iterated w/ SOLR for categorical filtering of multimedia objects
o Degrades gracefully w/o JS
o 508 compliant
· Content Workflow improvement
o Enabled some drag and drop setup
o helped w/ slideshow
o System to help manage multimedia usage consistency
o leads to better 508 compliance
· Akamaii Integration
o Clears cache automatically on node changes
o Push button purge
o Admin lets you clear a URL
· Fault Tolerance
o Developed a series of failure plans: what can break, what happens
o Went around environment and turned off things: servers, etc.
o Determine what happens and how to recover
· Need to load test iteratively
o i.e. a certain SQL approach was fine in dev but failed under load.
· Launch prep
o Conception to launch in 3 months
o Daily meetings
o Plan for background investigations
o Security tests for core code
o Intrusion testing
o 900 page document w/ multi levels of approval
· Launch
o Sat Oct 24, 1pm. Locked launch time 4 hours prior
o Automated transition happened right on time. Not a delay like a DNS change wait and see
· Numbers
o 100's of thousands uniques
o million page views daily
o 10's of mil per month
o 15k web submits per day
· 2 data centers
o Production
o Emergency
o Well over 60 servers
o Redhat Enterprise Linux
o Apache, memcache, mysql, SOLR, akamaii - pretty traditional
o Servers running virtual / cloud hosting
o RHEL
o Hardened to NSA guidelines
o Provisioned with a tool called Puppet
§ Provides data center automation
§ Recipe / model driven framework
§ Ensures consistency, repeatability & compliance
o CDN: Content delivery network
§ Akamaii SiteAccelerator, NetStorage & Livestream
§ Drupal integrated w/ Akamaii Cache Control utility
· Less than 10% of traffic hits origin servers
o Web
§ deduced private edit servers
§ Drupal 6
§ community / Pressflow patches
§ Supports DB replication
§ Full R/W splitting
§ Shared file system
o Cache Layer
§ Memcache & Drupal memcache
§ Cluster
§ Allows nodes to fail but still remain cache
o Server
§ Engine X tunning in front
o Database
§ RAM based filesystem for temp tables
§ Performance optimizations++
o Replication
§ Master (active) / Master (passive)
· Monitoring
o MySQL Enterprise monitor
o Nagios - infrastructure monitoring. swap space, free mem, proc usage. Has Drupal plugin i.e. if cron hasn't run
o Cacti
· Replication Monitoring
o Constantly updates the pool of active slaves
o Monitors all active slaves
o Removes slave f MySQL or replication is down
o Reinitializes when replication fails
o Adds slave back into pool when healthy
o Manages the replication hierarchy
· Environmental Sync
o Sync static assets up and down to NetStorage
o Sync virtual hosts and files to Disaster Recovery (DR)
o Replication of search index to DR
o Replication of DB to DR
· Hardware Scaling
o Quickly scales horizontally: add more web servers, DB servers, etc.
o Puppet handles all of the provisioning details
o Minutes from provisioning to handling traffic
· Data Scaling
o Core content
o High volume web form submission
o Allow for quick partial recovery
o Should be applied to other high volume data
· Development Process Scaling
o Multi vendors working many paths
o Individual Dev sites
o Branch per issue
o Branch per release
o Subversion / Git
· Release process
o Branch / integration sites
o Full featured but reduced staging environment
o At least one deployment per week, sometimes more
o Tool support (Jiira, Fisheye/Crucible for online code reviews)
· Since Launch?
o President's SAVE Awards
o Visitor Records
o Mobile version of sites / iPhone app / HTML 5 version of site with streaming video
· What Comes Next?
o How to scale user authentication?
o Handling user created content? Over 2 million web form submits since launch
- peege's blog
- Login to post comments
-
