D3 Tree

Forget Me Not


The picture is something I got from a florals postcard art collection from the late 1800s. The creator is unknown. I like the antique feel of it an I think it goes well with the tree, establishing the idea that the lineage in the tree flowed from this woman.

This tree is done in D3.js. I liked the look of it. It has a feminine quality that reminded me of fertility that came out of experimenting with the styling of the diagonals linking the nodes on the graph. The Tree graphic is hosted on the page in an iframe and is not yet responsive. The tree data is stored in a JSON file retrieved by D3.s json method.

This piece is is part of my experiments with data presentations as art.

Posted in Articles, DDJ, opinion, Technology | Tagged , , , | Leave a comment

Fat Arrow Functions

In ES6 it is common to see fat arrow functions. This way of writing functions has a syntax like this:
const MyFunction = () => { return 1;}
That is equivalent to:
Function MyFunction() { return 1; }
You’d call such a function like this:
MyFunction();

Functions created with the arrow syntax have the feature that the lexical “this” is remembered between invocations. You do not have to declare some extra variable to hold a consistent “this”.

Also, “yield” does not work in fat arrow functions so you cannot use them as generators.

Most likely you’ll want to use fat arrow functions in lambdas (functions that return immediately) and for functions that need to work with parent scoped “this.”

Posted in Articles, DDJ, Lrw updates, Technology | Tagged , | Leave a comment

Javascript Iterators and Iterables

The EcmaScript2015 standard specifies Javascript iterators and iterables. These specifications are protocols rather than builtins or syntax. You can implement the protocols in any object just by respecting some conventions.

Iterators:

Iterators are objects that know how to access items in a collection while keeping track of its current position in the collection. The object will have a next() method that returns the next item in the sequence as values and a boolean done flag. If the done field is true, the value field can be omitted and should be ignored.

Iterables:

Iterables are objects that implement the @@iterator method. This signature is available as Symbol.iterator. The @@iterator method must take zero arguments and returns an iterator object, as defined above.

Some iterators are also iterables.

Here is some sample code of iterables:
var myString = "do it";
console.log( typeof( myString[Symbol.iterator]) );

var iterator = myString[Symbol.iterator]();
console.log( iterator + '' ); // "[object String Iterator]"

console.log(iterator.next()); // { value: "d", done: false }
console.log(iterator.next()); // { value: "0", done: false }
console.log(iterator.next()); // { value: " ", done: false }
console.log(iterator.next()); // { value: "i", done: false }
console.log(iterator.next()); // { value: "t", done: false }
console.log(iterator.next()); // { value: undefined, done: true }

console.log( [...myString] ); // spread operator
console.log( "=======");

var myArray = [1, 2, 3, 4, 5];
console.log( typeof( myArray[Symbol.iterator]) );

var Aiterator = myArray[Symbol.iterator]();
console.log( Aiterator + '' ); // "[object String Iterator]"

console.log(Aiterator.next()); // { value: "d", done: false }
console.log(Aiterator.next()); // { value: "0", done: false }
console.log(Aiterator.next()); // { value: " ", done: false }
console.log(Aiterator.next()); // { value: "i", done: false }
console.log(Aiterator.next()); // { value: "t", done: false }
console.log(Aiterator.next()); // { value: undefined, done: true }

console.log( [...myArray] ); // spread operator

Some of my posts will detail patterns and functionalities that call for iterables.

Posted in Articles, Technology | Leave a comment

Promises

Promises are a good way to handle asynchronous functionality in javascript.
That asynchronous behavior makes a single thread of javascript run at a
furious pace with apparent parallelisms without all of the fuss of threaded
code.

In years gone past, asynchronous was handled by callbacks. You’d call an
asynchronous function, such as an ajax or other IO request, with a
callback. The callback would be called when the IO request completed.
That produced good results allowing the javascript thread to continue on
while the IO request was being processed.

Problems with that occur when async functions needs to called and completed
in an serially ordered fashion. The result would be code like this:
var data = ajax1( 'http://...', function(data, err) {
if( !err ) {
return ajax2( 'http://...', function( data, err ) {
if( !err ) {
return ajax3('http://...', function(data, err) {
// etc.
});
}
});
}
});

The nesting of functions within functions can get very deep and confusing.
The use of promises can clean up that code. In this way of programming,
each function called returns a particular instance of a Promise object.
The Promise object is instantiated with two arguments, resolve and reject.
eg:
new Promise( resolve, reject);

This promise object is returned synchronously and it will have one of several states: resolved, rejected or pending. Once a promise is either resolved or rejected is is deemed to be settled. A settled object can not change state back to pending or switch between resolved and rejected.

Another key characteristic of a Promise is that is has a standards compliant then() method. The then method looks like this:
promise.then(
onFullfilled?: Function,
onRejected?: Function
) => Promise

Both onFullfilled and onRejected are optional and will be ignored if they
are not functions. onFulfilled() will be called after the promise is
fulfilled, with the promise’s value as the first argument. onRejected will
be called if the promise is rejected and the first argument will be the
reason for the rejection. the attached then() can be called numerous times
on the same promise and each time then() must return a new promise. These
allow promise agregation and chaining, eg:
bringMe( URL )
.then(striptags)
.then(saveInFile)
.catch(handleErrors);

These chains can get complex.

I have also found the use of Promises.all() to be useful. It takes an array
or other iterable of promise returning functions and it returns a promise
that resolves when ALL of the iterable functions have resolve or upon the
first rejection.
var http = require('http');

function myGet(h, p) {
return new Promise(function(resolve, reject) {
// do a thing, possibly async, then…
http.get({ host: h, path: p }, function(response) {
var body = '';
response.on('data', function(d) {
body += d;
});
response.on('end', function() {
// console.log( "h: " + h + ", p: "+ p +"\n\n"+ body );
resolve(body)
});
});
});
}

var Uspecs = [
{h: "lrw.net", p: "/" },
{h: "lrw.net", p: "/learn-mern-part-1/" },
{h: "blog.triblive.com", p: "/" }
];
// proms is a promise that resolves or rejects after all have done so.
// the error message in catch() is only the first rejection. There could be others
var proms = Promise.all( Uspecs.map( function( el ) {return myGet( el.h, el.p);} ) )
.then(values => {console.log( "values : [" + values + "]/values " );} )
.catch(reason => {console.log("error occured" + reason )} );

console.log( "DONE" ); // prints out before any URLs are finished

There is a lot written about promises and other async handling patterns. It
can be useful to google javascript promises and look at some of that.

Posted in Articles, DDJ, Technology | Leave a comment

Learn Mern, React First – part 1

Learn Mern, React First

react-logo

I already have a working knowledge of MongoDB, Node and Express but not React. In order to refer to myself as MERN stack enabled, I’ve needed to learn React. I also want to be able to develop phone apps and React-native is touted as being excellent for that so I set out to seriously learn react from the ground up.

It has been a big jump to learn to use React and Redux for constructing web frontends. ES6 is a good part of the reason there is much to learn, but there is also JSX, a react way to combine HTML and javascript.

React prefers the use of ES6, which is the next generation of javascript. I am told that you can use ES5 (the more widely known version of javascript), but I have not seen much in the way of learning materials for learning React with ES5 only.

ES6 is available now in the most current editions of widely used browsers and support is good in node. There are transpilers that allow ES6 to be compiled down into ES5 so React written in ES6 can run on older browsers.

I looked around at various books, websites and videos in order to find something that would allow me to get up to speed in React. I came to this task with a good working knowledge of ES5, Express, Node and Npm. It seems like installing almost any front-end packages or tool chains requires some Node and React is not different. I settled on Stephen Grider’s Udemy.com course called Modern React With Redux.

That course involved about 150 video segments totaling over 22 hours. However, you can’t sit down and expect to go through the course in 22 hours of video watching. You need to work through code yourself while following the videos. That means a lot of pausing the video while you switch to your code editor to duplicate what Grider is doing.

It took me about 75 hours of screen time to get through the course. You could easily do that in 2 weeks of full time effort, but I have a day job so I did about 2 hours per day and it took 46 days.

Grider is verbose in his approach to teaching and I found that extremely helpful. He constantly re-summarized and developed the material very thoroughly. I recommend the course.

But, this course really only gets you React and Redux. It shows how to hook onto some commercial APIs like Youtube. But if you want to get the rest of the MERN stack, you’ll have to look elsewhere. For that I am currently engaged with Alex Paterson’s “React Native with An Express/MongoDB Backend.”

Paterson’s course is extremely Mac centric. But I am after the React-native elements, anyway, in order to do phone apps and a Mac is essential for doing iPhone apps. So, I bought a Mac Mini for doing this course. I am such an open source snob that buying a Mac was like pulling teeth! But my heuristic is that I will not use closed source unless there is no reasonable open source alternative.

In addition, Paterson’s course moves at a much more brisk pace than the Grider course. There is much less molly-coddling and hand-holding in Paterson’s videos. I’m only half way through this course, so I will reserve judgement until I get all the way through, though I can says that so far I am learning the material.

 

Posted in Articles, DDJ, Technology | Comments Off on Learn Mern, React First – part 1

MEAN vs MERN

The Mean vs Mern stack decision for me has a couple parameters. I would like to choose between Angular and React based on which is most likely to bring the greatest return on learning time invested. These two both involve getting the most current javascript skills which I would do anyway.

The market of jobs, even jobs that are not consulting jobs, gives a good snapshot of the local demand. There is information on the size of the local market, but that does not offer an idea of ow supply and demand are balanced nor of over trends.
The days that jobs remain open is an available data point that does speak to the supply and demand. I compared mean/mern stack markets with WordPress in order to have a bit of context.

Here are some Pittsburgh figures:

Technology Market Size Ave. days open
MongoDB 28 27
Nodejs 136 20.15
Reactjs 20 21.7
Angularjs 83 17
WordPress 10 13.3

In comparing the opportunities, the size of the market is important and the larger the market the better. However, the most important variable is the number of days that jobs remain open. The more of those, the harder it is for prospective employers to fill their roles and the greater the opportunity there is for consultants.

Here is a trend chart for Google searches with respect to Angular.js, React.js, Nade.js and MongoDB. You can see that Angular has a lion’s share of the attention, but is falling off. In fact it fell from 100 last summer to 72 now, a loss of 28%. None of the other technologies has lost as much of a share of attention.

My inclination:
Given the supply and demand balance in my area, as well as the national search trend, I think React.js is a better bet than Angular.js, so I am leaning toward a MERN stack.

Posted in Technology | Tagged , | Comments Off on MEAN vs MERN

Learning MeteorJS

guage-screenshotThis article looks into my experience learning the MeteorJS Application Framework and doing a production project based on that. The article should be useful to you if you are trying to decide whether to invest the time and effort needed to learn Meteor and it could also help you determine whether Meteor is a suitable framework to use on your own projects.

Meteor has a mission of providing a simplified way to produce protypes in a
day or two and production applications in a couple of weeks. I do see Meteor
as a browser/server or mobile app/server product rather than a desktop application.
Meteor is built from a number of subprojects and exisiting codes. It uses an
isomorphic approach to provide both client and server code in the same file.
Javascript is used to drive both the frontend and the backend. There is a
templating language used for the HTML output.

On the server side, Meteor uses mongoDB from a nodeJS server. If you haven’t used mongoDB, you need to know it is an incarnation of the current trend toward nosql database systems. It is a document database rather than a tradition table/row/field style accesed by SQL. The documents are in a sort of a dialect so similar to JSON that they can be directly accessed in the javascript on both the backend and the front. mongo is also very speedy and scaleable.

There is a significant community involved in Meteor and there are a growing number of excellent plugins. There is an official tutorial on the Meteor.com site as well as documentation. There are also blogs, books and other resources to help get an aspiring Meteor developer started.

My own attraction to investigate Meteor had to do with my job in a newsroom at a mainstream daily newspaper. Part of what I do is data driven journalism and that involves creating interactive visualizations on very tight deadlines. I need something that gets a decent quality visualization up really fast. I have also been dealing a lot with maps and in doing that I have found that geojson is a great goespatial code to use. The geojson feature objects can be used in mongodb to create searches that are not available at all in most database. For example, given a geographic coordinate pair, I can search on mongodb for the city, county or state that contains that point.

I heard that Meteor allows really fast development of this sort of mongoDB applications. I tend to shy away from frameworks generally as they tend to obfuscate things I need access to and the can make it really convienent for me to not learn basic sorts of things I need to know. I admit, I am lazy. I do not want unneeded cognitive load. But Meteor seemed so promising that I wanted to try it out.

Learning Meteor

In order to use Meteor, you need to have at least a modest amount of javascript knowledge as well as fairly solid HTML and CSS knowledge. The server side code is going to be easier to understand if you have a grounding in nodeJS and mongoDB.

I came to Meteor with an advanced knowledge of Javascript and a few months of mongoDB learning via project development. My process to learn Meteor involved doing the tutorial on the Meteor site and looking a some projects done in Meteor. I have about 2 hours per day set aside for learning or developing new technolgies. I invested about 2 weeks of that time in Meteor before I started my own project.

Doing this first production project in Meteor took about 25 hours. I can see that doing additional projects will be considerably faster. I had to stumble along especially in integrating Meteor with leaflet and to get http queries to work.

Project details

The USGS offers hydrological information from a REST api. There are 1000’s of guages all over the US for rivers, lakes and groundwater. The API that I used allowed the extraction of current data for specific guages. I used an array of guage ID numbers to send off the http requests.

The data returned by the USGS comes back in a JSON format. My serverside code parses the incoming USGS data and I built an upsert query to deposit the data into mongoDB. If you are new to noSQL databases, and upsert is a combination of insert and update. If the record exists, it is updated, otherwise it is created.

Meteor monitors the activity in the database, apparently by watching the log. When a datum is modified, ‘subscribers’ to that datum at notified. This notification extends to the frontend clients. There is a websockets implementation that carries the notifications out to the client. The data modification messages can go both ways, such that when data is modified on the client, the server side code and database is updated accordingly which in turn causes all other clients to receive notifications.

In order to integrate with leaflet, I used a package identified as mrt:leaflet-0.6.4.This worked well in getting my intial map. In order to update the map’s markers and popups in real time I needed to ‘subscribe’ to certain data messages and then act on the received information.

Meteor takes a whole lot of the tedium out of coding an application. It not only produces a great skeleton at the outset of a new project, there are lots of very useful facilities baked into the product. In addition to the server and client side codes, the templating is easy to use for data pulled out of the database. There is advice among data visualization people that you should concentrate on presentation first thing into the project. Meteor allowed me to do that.

The way the code handles client and server side communication is great. The deloyment to production was a real breeze. I was able to even bring up a smart phone app on an emulator that was a real match to the web app. All that code is already done via an integration with Cordova. In fact, the way this code can bang out iOS, Android and web apps from the same code base is a real boon in my world of very pressing deadlines and breaking news.

Deployment of my first application went to the Meteor site. It was incredible smooth and easy. I am not sure how this hosting can scale though the docs do say you can aim your own domain name at the hosted app. There is both free and premium hosting at openshift.com. There is also hosting on Digital Ocean starting at $5 per month. An installer call ‘up’ makes it easy to do the installations.

Conclusions

Meteor does provide a great deal of what I need in terms of fast running, scalable applications. It also hits my need for fast to deploy and production quality. The community has provided great plugins that I can use to help out with integrating stuff into Meteor. I found the plugin process easy to use. In fact the whole thing has been quite satisfactory so far.

Posted in Articles, Technology | Tagged , , , , , , , , , , | Comments Off on Learning MeteorJS

Google Drive for WordPress

The integration of Google Drive for WordPress offers some great features.  A plugin for WordPress called docs to WP connects Google Drive to WordPress, allowing a person to create post and edit posts on a Google Drive account.

If you’ve never used Google Drive or its predecessor, Google Docs, I’ll describe it for you.

Image representing Google Docs as depicted in ...

Drive is like have an online, private disk drive that holds 5 Gigabytes of date. Various online applications allow you to do wordprocessing, spreadsheets, presentation slides, drawings and so on.

It allows an author to access work from just about any computer, pad or smartphone with an internet connection. The files can also be shared with others in order to do collaborations. I think it is a better way to author work that to use the WordPress editor. I like to collect a pile of source documents and URLs related to what I write.

I also like to upload my stuff into WordPress where I can run the SEO optimization tools along with content enhancers like Zemanta. This allows me to put up stories from anywhere on the web.

I have found that images I insert into a Google Drive document do not make it onto the WordPress site. The need to be added separately once the text is uploaded.

The way my set up works is that I write my copy in one of several folders. When I am ready to upload it to WordPress, I just move it to a folder I named SendToWP.  Any folder name can be used as this is part of the configuration you add to wp_config.php. WordPress queries my folder once per minute at Google. If it finds a new file, that file is uploaded as a WordPress post without any intervention on my part. The Google Drive file is then moved to the Done directory. I find that useful.

This facility is one of a number of pieces that are useful to make WordPress more useful to news gathering people. But software like this tends to be used far beyond its original intended use. My hat is off to William P. Davis and his employer Bangor Daily News.

Technical Details:

The way the plugin works is that you install the plugin with the WordPress plugin installer. The plugin is at http://wordpress.org/extend/plugins/docs-to-wordpress/. Then you need to put some configuration text into your wp_config.php file. The updates to wp_config.php are shown at http://dev.bangordailynews.com/2011/06/20/quick-update-to-the-docs-to-wordpress-plugin/.

Screen Shot of plugin admin pageWhen I installed the plugin package docs-to-wordpress by using the plugin installer,it installed 3 plugins. The plugins include docs-to-WP, Docs to WordPress extender – Run on Cron, Docs to WordPress extender – clean content, strip comments.

I also added some code to the file docs-to-wordpress/extend-clean.php via the plugin editor.

This code allows you to put a Title in the text of the Google Drive file. You just need to put it first in the text and follow the titel with a pipe symbol, usually shfit+ on the keyboard. This code is is taken from the plugin author’s William Davis’s site, http://dev.bangordailynews.com/2011/12/22/setting-the-head-of-a-post-using-a-delimiter-in-your-doc/

/**** added code  ****/

/*

Plugin Name: Extend Docs to WP like so

*/

 

add_filter( ‘pre_docs_to_wp_insert’, ‘bdn_split_post’ );

function bdn_split_post( $post_array = array() ) {

 

$exploded_fields = explode( ‘|’, $post_array[ ‘post_content’ ] );

//Sometimes people forget a pipe, and we don’t want to put the entire post in the headline

if( is_array( $exploded_fields ) && count( $exploded_fields ) >= 2 ) {

 

//Save the old title in case you want to do something with it

$old_title = $post_array[ ‘post_title’ ];

 

//Set the title to the first occurance.

$post_array[ ‘post_title’ ] = strip_tags( $exploded_fields[ 0 ] );

//Unset the title

unset( $exploded_fields[ 0 ] );

//Now restore the post content and save it

$post_array[ ‘post_content’ ] = implode( ‘|’, $exploded_fields );

}

 

return $post_array;

 

}

/**** end of added code ****/

There are a number of other pieces that need to be integrated in order to produce what  believe is a truly great piece of software for professional newsrooms. I’ll talk about those in future posts.

Posted in Articles, Technology | Tagged , , , , , , , , , | Comments Off on Google Drive for WordPress

Websites for Mobile News

mobile devices

mobile devices (Photo credit: Hands On Support)

Websites for mobile news is a topic I’ve been working on lately. I’m pretty focused on news and technology now, as I am doing some consulting development for a newspaper. In the process of researching this topic I came across a report on the state of the media that involved a survey of mobile users. The link to the report is at the end of this article but here are some highlights:

Mobile news reading is an activity that is slanted towards younger people. Mobile readers are more likely than non-mobile users to use social media sites. More than half use 6 or more platforms to get local news and information. But, here’s a stickler: only 1 in 10 use a mobile app to get local news, even though 1/2 use mobile devices to get local news.

English: A pile of mobile devices including sm...

A pile of mobile devices including smart phones, tablets, laptops and ebok readers. (Photo credit: Wikipedia)

My comment on that is that I 100% agree with those who use the mobile. browser to get local news. I do that too, and I avoid sites that put up mobile only versions of their websites. I have found it very frustrating to use a site on my computer and then, while out and about, to check out the same site and see a fraction of the information. I want the whole enchilada. I want a mobile device that can display the real website even though I am over 60 and my eyes are not sharp. I want a site that handle enlarging the text and pictures. I’ve seen it done. I know it can work and I suspect everyone will be doing it.

Another thing about mobile users is that they are much more active news users than

the local news, locally

the local news, locally (Photo credit: McBeth)

non-mobile users.

Mobile useage tends to rise with income and it has a non-white skew. It is also more prone toward suburban users. The longer a persons tenure in their current community, the less they use mobile devices for getting news.

So, for general news sites my opinion is that efforts should be focused on making the web site work for mobile devices rather than spending a lot of resources on an app.

 

http://stateofthemedia.org/2011/mobile-survey/

LOADING...

Posted in Articles, Technology | Tagged , , , , , , , | Comments Off on Websites for Mobile News

Responsive Web Design

Responsive Web Design is another one of those ideas that has taken the web community by storm. Responsive Web Design, a.k.a.RWD, is a set of techniques that allows a web site to present content well to all are almost all users nor matter what sort of device they use. There has long been a problem associated with making a website that is usable by both a desktop computer with a high resolution monitor and a smart phone with a very smallscreen.

Web Frustrations of a Designer

Web Frustrations of a Designer (Photo credit: dailyinvention)

Perhaps you recall the days when sites had a disclaimer on the that went like: ‘This site is best viewed with X browser‘ and how annoying it was if you used some other browser. There was a time when 80% of website visits came from one or two browsers. That is not the case now and has not been for some time. Today, it is unacceptable to NOT support the most common browsers, even though it can be expensive and time consuming to test a website design across a large cross section of browsers.

There are many common browsers today operating on a lot of different devices. According to Yahoo’s Graded Browser Support pages, “There are over 10,000 browser brands, versions, and configurations and that number is growing.” Savy large scale web operation test their sites to insure that they support the most common browsers, but I haven’t heard of anyone who tries to support an exhaustive list of all or even nearly all browser.

InfoGraphic about widths of various displays

There are more problems to solve than just screen size. CSS is used in web designs to control decorations like fonts or placement of items on the screen and javascript is used to handle dynamic features of the user interface, like popup windows or folding menus. There are also proprietary or ‘3rd party’ capabilities that may not be present in a browser, such as flash or rendering of non-html/xhtml documents.

Some browsers, such search engines, do not support javascript or CSS and are really only in search of the content. There are also ‘accessibility’ browsers such as those that provide an audio interface. In some security conscious environments, javascript may be turned off completely, but CSS is left on. For this the notion of Progressive Enhancement has become popular among web designers. The idea is that the content will be available to any browser that understands html and those that understand CSS will have styles applied to make the experience better and those that use Javascript can have the full blown user experience. There is no switching the user from one site to another according to the browser that is detected.

A responsive web design page can be resized on the fly. The text will reformat itself and the images will grow or shrink according to the screen size. If CSS, Javascript of both are not available on the browser, the content will still be accessible.

Progressive Enhancement (PE) involves a particular sequence of design. First the basic markup, geared to the lowest common denominators of browsers, is applied to the content, then ehnhancements available in common, widely used browsers, are externally linked into the page. The external linkage means that enhancements that are not used on a page are not downloaded and that makes the page load quicker.

I’ve been experimenting with the Skeleton WordPress theme in order to implement RWD on this site.

Posted in Articles, Technology | Comments Off on Responsive Web Design