Hey folks, 

Thanks for a great session and cheers to the “ghost” voice that brought you all back to life halfway through. Keep in touch. http://twitter.com/buzzillions

Here’s the Jira Greenhopper presentation:

http://bit.ly/buzzjira

Good luck to JC and family in your move to Australia!

Cheers, Jim

This Endeca-centered blog post is the first in an occasional series where we’ll discuss how we’re using Endeca. One of our engineers was mentioning how finding Endeca tips using Google wasn’t very easy. So here goes.

We’ve got enough machines in various development and production environments that standardizing installations and being able to bring new machines online quickly has become a business necessity.

So during this upgrade to version 6.1, we opted to run the whole install from Puppet , an open source server provisioning software project. We already use it to provision most of the other major software in our infrastructure.

I generally tell people that it takes about 4 times as long to automate a process as it does to “just do it” manually. That’s a hard pill to swallow when time spent on infrastructure has to be balanced with time spent innovating. For us, Endeca has long been one of the installs that’s done by the software engineers and not the systems team. But upgrading from 5.1.3 to 6.1.1 was the right time to automate the whole thing. So we grabbed our wiki pages, our journal entries and telnet session logs and sent them to our brave but persistent systems admin. He figured out the test cases, asked questions and probably installed Endeca over 50 times on the way to success.

The good news is that he can get the whole thing from a blank machine to a machine running forge/dgidx and dgraphs that have yesterday’s data and config files in about 15 minutes using Puppet.

Note that it helps to have all necessary files in a code repository system that Puppet has access to. We use Subversion.

If you’ve ever installed Endeca, the steps are usually included in 3 to 6 different PDF files that are in various “docs” directories. Though we’re partial to the CommerceSuiteQuickStartGuide.pdf these days. The migration steps are where it gets complicated. You generally need to work with Support to find out which files are critical and which ones are unnecessary. The various Migration Guides are fairly incomplete for a real migration.

Here are our steps for Migration from 5.1.3 to 6.1. Note that these steps will be slightly different than your specific steps. Really, the hardest part about the automated install is to formalize the exact ordering and dependency of all the install elements and the migration elements. The way puppet works is that at each step, you need to specify a “test” so that Puppet doesn’t run a step that it has already run before (it might be the existence of a directory or a file or the response code from the emgr_update calls). I think the phrase is making an “idempotent operation”. Also note that we use Red Hat Linux 5.

1. Create a user account specific to the version of endeca you’re installing.
   •  Don’t just use a username that is generic. Using one that is specific per version allows you to upgrade on the same server since you may want to add new software to a server where you’re already using Endeca. The environment variables for different versions are different so we opted to have different users since our inittab scripts run as this special version specific endeca user.
   • Puppet creates the endeca version specific user along with its profile environment variables file.
2. Standard install steps
   • Create a shortcut UNIX alias to change directories to the base /endeca directory, call it $ENDECA.
   • Create a shortcut alias for your application name, call it $APPNAME
   • As the endeca version specific user,
     • Install the MDEX software using the silent install option. Easy. No config file needed.
     • Install the PlatformServices software using the silent install option. Here you need to decide whether it’s a FULL install (for Forge/Dgidx servers) or an AGENT ONLY install (MDEX/Dgraph only servers). Silent install file is simple.
     • Install the Merchandising Workbench software using the silent install option (or your appropriate Workbench package). Silent install file is simple.
     • Install the ContentAssembler API basically by unzipping into the $ENDECA
     • Install the DeploymentTemplate 3.0 using a silent install file. This silent install file can be quite complicated but we just use the simple options (anybody know where the documentation for the advanced options is located?). NOTE that since Endeca uses relative paths during deployment deploy, we had to write a shell script to call their shell script to get these relative paths to work in Puppet. The other installs seem to use the path that you pass in.
3. Migration steps
   • State files
     • To keep your N values stable between instances, you need to grab the two files located in $ENDECA/$APPNAME/data/state and move them to your new deployment application in the same directory. We keep update these files in our Subversion repository after every Forge run.
       • autogen_dimensions.xml.external.gz
       • autogen_dimensions.xml
   • Pipeline files
     • The pipeline is divided generally into two parts: Dev Studio files and Web Studio files.
       • The Dev Studio files are files that users DON’T change in Web Studio. They are generally located in $ENDECA/$APPNAME/config/pipeline and you can just copy them to the new deployment application in the same directory.
       • The Web Studio files are kept in the “ether” so you have to extract them using the “emgr_update” application. You set which files are “Web Studio maintained” by changing the “ConfigUpdate” section of the AppConfig.xml. Make an empty directory $ENDECA/$APPNAME/temp and extract the files using the emgr_update command and then copy to the pipeline directory in the new deployment application.
         • emgr_update –host localhost:8006 –action get_ws_settings –app_name $APPNAME –prefix $APPNAME –dir $ENDECA/$APPNAME/temp
       • Once you’ve copied the Dev Studio files and then the Web Studio maintained files to the new pipeline then you can stuff the files into the new Web Studio.
         • emgr_update –host localhost:8006 –action update_mgr_settings –app_name $APPNAME –prefix $APPNAME –dir $ENDECA/$APPNAME/config/pipeline
         • If you have externally managed dimensions, you will need to ignore a warning like “warning, cannot find <file>, proceed anyway?” . Just say yes, it doesn’t actually need the externally managed dimension files.
   • Data files
     • In preparation for the baseline update, we move our data files to the $ENDECA/$APPNAME/test-data/baseline directory. We mainly use XML files as our pipeline source files.
   • Web Studio usernames and preferences
     • Copy the “webstudio” directory in the 5.1.3 $ENDECA_CONF/state
     • Remove the existing “webstudio” directory in your new deployment application in $ENDECA_TOOLS_CONF/state
     • Turn off the Endeca processes. Paste the “webstudio” directory from above into $ENDECA_TOOLS_CONF/state. Turn Endeca processes back on.
   • Before you baseline, here are some miscellany:
     • By hand, migrate your AppConfig.xml from 5.1.3 to 6.1.1 (not many material changes)
     • In AppConfig.xml, change “pipeline.epx” reference to “Pipeline.epx”
4. Run the $ENDECA/$APPNAME/control/initialize_services.sh to setup the servers and modules explained in the AppConfig.xml
5. Run $ENDECA/$APPNAME/control/load_baseline_test_data.sh
6. Run $ENDECA/$APPNAME/control/baseline_update.sh
7. At this point, via Puppet, with NO human intervention we have installed a new Endeca master server (Forge/Dgidx). The Puppet installation for an MDEX/Dgraph server is just the user creation, MDEX install, PlatformServices AGENT ONLY install and an inittab script. Then you start the process and include the server and port into the AppConfig.xml which will configure the server the next time you run any EAC command. 

Send us your comments and hopefully other Endeca users will start to blog/write about the ways they make life easier on themselves or the cool things they do. The Encyclopedia (read “wiki”) part of the Endeca documentation system is also a great place to make updates and share tips.

Cheers, The Buzzillions Team

The Buzzillions Team

New Television page. Here’s the screen of the OLD Televisions page:

The other day I was in Target looking for air beds for a visitor coming in town.  While it’s easy these days to find product reviews on things like digital cameras and televisions, it’s a little trickier when you’re looking for other products.  As I stood there blinking at the choices, (varying price ranges, familiar and unfamiliar manufacturers etc.) I thought…if only there was a way I could get some advice here.  Did I need the $100 air bed with the foot pump, or would the $39 one do just fine?  While it was tempting to go home, get on the computer and see what Buzzillions had to say about it, I ended up choosing blindly.  Now I (and YOU) don’t have to.  With our new Buzzillions iPhone app, you can (if you have one of these handy phones OR the ipod touch) download the app and be on your way to having millions of product reviews at your fingertips.

So the next time you’re at a retailer, scratching your head as you mull over the options, you can whip out your iPhone, type in the product name and get a little guidance from reviewers like you.  Best of all…it’s FREE. We’d love to hear what you have to say about it, things you like, things you don’t, so we can work to improve the application to make it even more useful.

I know we’re biased, but Buzzillions.com is the best place for top rated products with ratings from other parents like us!  Buzzillions.com allows us to read reviews from real parents on everything from the best baby bottles and the most reliable baby diapers to trendy gift ideas for our 3-year-old’s upcoming Birthday.

However, sometimes we need specific answers to questions and don’t have time for in-depth research.  That’s where AnswerBox comes in.  What do other parents think about using BPA-free baby bottles and which ones do they recommend?   What are the best sources for “official” car seat and booster seat safety information?  How well do gDiapers work?  We love the fact that the inserts are flushable and compostable, but do the reusable diaper shells really keep it all in (yes, they do)?

As you can see, we’ve got a lot of questions.  The local baby boutique is good for some answers, but who has the time these days and how do we know that they aren’t just trying to get us to buy their products?  Baby books are great for one-way information, but not specific answers.  There must be a lot of other parents out there with answers!

With this in mind, PowerReviews created AnswerBox.  Our newest feature enables your customers to ask questions about specific products and tap into a large online community of active shoppers for answers.

Actually, that image came from Jim in the office, taken on a sub-par camera phone for a mousepad
on Allsop. No shame in bootstrapping the feature, right? That was August of 2006. By now, thousands of customer images have come in, and I’ve graduated from intern status. Fortunately, I will admit that all of the images look better than Jim’s hand in the first customer image. Here’s a recent example from a review on the Nikon D200:

We’d love for you to try out the feature yourself. Perhaps you already own a product from one of the following popular retailers that use Customer Images. If so, share your experience with the product by writing a review and be sure to attach an image while you’re at it!

• Adorama
• Netshops
• Evogear
• Ritz Interactive

(First of all, we AVOID as many Web 1.0 lame-isms as possible but you never know when an 800 pound Gorilla reference is going to delight the noobie who’s still sending his friends joke emails.)

Okay back to tech talk…so we go for the “best of breed” approach and generally operate on the bleeding edge. Why not choose the tried and true tech of yesteryear? Well, using the right combo of stable open source and innovative enterprise software gives us a legion of low-cost outsourced programmers. Why reinvent the wheel? (…when you can just use other people’s cliches…)

Whoever said “Work smarter not harder” clearly did NOT work at a startup.

When you’ve got to work harder AND smarter, you want the best tools at your disposal and you want to dispose of the tools that don’t cut the mustard. We buy some tools (e.g., Jira, Intellij, Endeca) and use open source for others (e.g., Hibernate, EHCache, Apache, Tomcat, Spring).

We even tested three wikis for a month until we chose the one we wanted to use (Twiki). There were several Wikis that didn’t even support attachments…weird. As for why wikis have two search boxes…well, that’s a mystery for the ages.

If you add in the individual productivity tools such as Excel, Powerpoint – does it really boost productivity , SnagIt, Google Docs, Mind Manager and others, the list of tech becomes staggering.

Stay tuned and we here at Buzzillions will open the hood so you can learn from our mistakes and in turn share your insights with us.

Can you add to this list of Web 1.0 lame-isms?

“800 pound gorilla” – I later found out that a VERY LARGE gorilla is generally 550 pounds so if you’re the 800 pound gorilla you can really kick butt (unless you’re slow or dumb I guess)

“how the sausage gets made”- I researched this and wish I hadn’t

“dog and pony show” – many dotcom road shows were just a substantive

who’s got more of these?