Pipelines on TSO

Today for the first time the open sourced Pipelines processor for NetRexx ran on TSO. After NetRexx 3.03 did not run on OMVS at all due to codepage problems, and after the integrated Pipelines suffered from a JVM bytecode mismatch problem, I have incorporated the Pipelines build into the Ant build for NetRexx and finally have bytecode for JVM 1.5 for the Pipelines classes. So with pleasure I announce the first VM/CMS compatible version of Pipelines for NetRexx running on MVS/TSO. In a few moments I am going to run it on real iron (my employer's EC12) to get an impression of its performance. After some work on the manual it is going to be released together with NetRexx 3.04.

NetRexx Pipelines

I am working on integration of Pipelines (by Ed Tomlinson and Jeff Hennick) with the NetRexx translator for release 3.04, that will be available somewhere in September 2014. Prompted by fairly unrelated thoughts about Java 8's lambda's and the pipelines on collections classes made possible by those, I remembered how closely Rexx is to Pipes on VM/CMS (as is my conclusion based on lots of stories) I was looking into a way of making the use of a pipeline as straightforward as possible. In an earlier stage (no pun intended) I had repackaged the njpipes package into using the org.netrexx.njpipes hierarchy (where previous releases used pipes. and stages. packages) and was not entirely happy with the visual distraction the elaboration of the package names (to adhere to the package naming conventions) introduced; while an the other hand feedback from early adapters of the njpipes package indicated that using CLASSPATH and successfully installing the package was still problematic, undoubtedly because people trying the pipelines would be more proficient with Rexx than with Java. The solution seems to be to integrate the package with the NetRexx translation archive itself, making sure there is not import needed to use a pipeline. The pipes compiler compiles a NetRexx source file which includes a pipe specification to a .class file - even if it contains no NetRexx but only a pipe - or a number of pipes. At the moment everything works and the test cases work. I am working on the manual and try to get it into a shape that I can send to John Hartmann, who earlier on promised to write a short introduction. When it all works, it will also be the first time an open source package will provide a pipes implementation for z/OS TSO.

NetRexx, Raspberry Pi and MQTT

It is more affordable, easier and quicker than ever to acquire, install and use a system that would still have cost a lot of time and money some years ago. A $35 Raspberry Pi, a card-size computer (bring your own keyboard and screen, until ssh is enabled by default by the distributions) and a message broker running on it can do impressive things. I am toying around with some security cameras, only, seeing the ease with which they can be compromised, I do not run them exposed to the internet, just use a Raspberry Pi for tunneling their http over ssh, and some software to control them, capture events and send them to a message broker that runs overseas, so I can have a good near-realtime view over what happens. As an indication how quickly the developments are, I had to revise a chapter in the NetRexx Programmer's Guide, that was written after the 3.01 release, already twice for the impending 3.02 release. The version of Linux running on the Raspberry, aptly named 'Raspbian' because it is a Debian for the Raspberry, needed to be compiled with the soft-float api because the Oracle Embedded Java version required it - no longer with the preview for version 8. This prompted me to refresh the install of one of my two Raspberries. I am happy to be able to state here that it took less time than ever to refresh an OS and my whole development environment. This is due to at least three facts: 1) The OS images are ready to run and only have to be copied to a SDHC card - I use SanDisk Ultra 16GB cards which are cheap and run well with the Raspi. 2) The instructions provided by nice people on their blogs are excellent. Oracle's VM still runs rings around the others, speed-wise, so this was an easy choice. I used this one: http://www.savagehomeautomation.com/projects/raspberry-pi-installing-oracle-java-se-8-with-javafx-develop.html which is good and to the point. You can skip the GUI instructions for writing the SD card and just use dd - but I presume you already did if you are in this stage. The JDK is installed like this: scp it to the raspi sudo mkdir -p -v /opt/java
tar xvzf ~/jdk-8-ea-b36e-linux-arm-hflt-29_nov_2012.tar.gz (the filename changed)
sudo mv -v ~/jdk1.8.0 /opt/java
sudo update-alternatives --install "/usr/bin/java" "java" "/opt/java/jdk1.8.0/bin/java" 1
sudo update-alternatives --set java /opt/java/jdk1.8.0/bin/java Actually, it runs if you only run the tar step, but the added tidiness in telling apt that you are running this is excellent. 3) I have all my stuff in Git so I can easily move it everywhere I am. Installing mosquitto For mosquitto, the open source mqtt broker (at http://http://mosquitto.org), there is also a debian package (only for the hard-float Raspbian, another reason to move to this, having previously run Java 7 on the soft-float Wheezy distribution). This boils down to (on the raspi): wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key
sudo apt-key add mosquitto-repo.gpg.key
cd /etc/apt/sources.list.d/
sudo wget http://repo.mosquitto.org/debian/mosquitto-repo.list
sudo apt-get update
sudo apt-get install mosquitto
That is it - a message broker, programmable in NetRexx (on the Java Runtime) for $35, an SDcard and half an hour of work. The new NetRexx version 3.02 has examples on how to use mqtt (and also WebSphere MQ, to which it connects, by the way). The examples are short, so I can publish them here:

import java.sql.Timestamp
import org.eclipse.paho.client.mqttv3.

class Publish implements MqttCallback
  
  method Publish()
    conOpt    = MqttConnectOptions()
    conOpt.setCleanSession(0)
    tmpDir    = System.getProperty("java.io.tmpdir")
    dataStore = MqttDefaultFilePersistence(tmpDir)
    clientId  = MqttClient.generateClientId()
    topicName = "/world"
    payload   = "hello".toString().getBytes()
    qos        = 2    

    do
      broker  = "localhost"
      port  = "1883"
      brokerUrl  = "tcp://"broker":"port
      client  = MqttClient(brokerUrl,clientId, dataStore)
      client.setCallback(this)
    catch e=mqttException
      say e.getMessage()
      e.printStackTrace()
    end -- do

    client.connect()
    log("Connected to "brokerUrl" with client ID "client.getClientId())

    -- Get an instance of the topic
    topic = client.getTopic(topicName)
    
    message = MqttMessage(payload)
    message.setQos(qos)
  
    -- Publish the message
    time = Timestamp(System.currentTimeMillis()).toString()
    log('Publishing at: 'time' to topic "'topicName'" with qos 'qos)
    token = topic.publish(message)
    
    -- Wait until the message has been delivered to the server
    token.waitForCompletion()
    
    -- Disconnect the client
    client.disconnect()
    log("Disconnected")
  

  method log(line)
    say line
    
  method messageArrived(t=MqttTopic,m=MqttMessage)
    log("Message Arrived: " t m)
    
  method deliveryComplete(t=MqttDeliveryToken)
    log("Delivery Complete: " t)

  method connectionLost(t=Throwable)
    log("Connection Lost:" t.getMessage())

  method main(args=String[]) static
    Publish()

And the corresponding Subscriber is like this:

import java.sql.Timestamp
import org.eclipse.paho.client.mqttv3.




class Subscribe implements MqttCallback




  properties
  client = MqttClient
  conOpt    = MqttConnectOptions()
  tmpDir    = System.getProperty("java.io.tmpdir")
  clientId  = MqttClient.generateClientId()
  topicName = "/world"
  qos        = 2    
  
  method Subscribe()
    do
      connectAndSubscribe()
    catch mqx=MqttException
      log(mqx.getMessage())
    end
    -- Block until Enter is pressed
    log("Press <Enter> to exit");
    do
      System.in.read()
    catch IOException
    end
    
    -- Disconnect the client
    client.disconnect()
    log("Disconnected")




  method connectAndSubscribe() signals MqttSecurityException,MqttException,MqttPersistenceException
    conOpt.setCleanSession(1)
    dataStore = MqttDefaultFilePersistence(tmpDir)
    do
      broker    = "localhost"
      port    = "1883"
      brokerUrl    = "tcp://"broker":"port
      client = MqttClient(brokerUrl,clientId, dataStore)
      client.setCallback(this)
    catch e=mqttException
      say e.getMessage()
      e.printStackTrace()
    end -- do
    
    this.client.connect()
    log("Connected to "brokerUrl" with client ID "client.getClientId())
    
    -- Subscribe to the topic
    log('Subscribing to topic "'topicName'" qos 'qos)
    this.client.subscribe(topicName, qos)




  method log(line)
    say line
    
  method messageArrived(t=MqttTopic,m=MqttMessage)
    log("Message Arrived: " t m)
    
  method deliveryComplete(t=MqttDeliveryToken)
    log("Delivery Complete: " t)




  method connectionLost(t=Throwable)
    do
    connectAndSubscribe()
    catch mqx=MqttException
      log(mqx.getMessage())
    end
    
  method main(args=String[]) static
    Subscribe()

Both work with the current Paho Java client library, to be gotten at: http://www.eclipse.org/paho/

NetRexx Open Source

Today was a very good day for Rexx, as it is the day that NetRexx was open sourced by IBM and the Rexx Language Association. More specifically, RexxLA is taking over from IBM. This open sourcing process has not been a short one, and there is no real reason to go into specifics about where it got stuck and who pulled it out of the mud, various times. There has been some incredible work done by some people at IBM - I am not really sure if they want to remain nameless, but I have an inkling that IBM sees it that way. I will remember them anyway and suggest they be honorary members for life of RexxLA.

It is no secret that the economic crisis and cutting cost played a part in this prolonged process. Also, the nefarious Linux lawsuit against IBM, by - I forget their name already - ah no, it was SCO, which took advantage in the vilest possible way of the OS/2 and AIX filesystem that IBM donated to Linux open source, also made IBM wary and insured that any Open Sourcing will be under close scrutiny and a watchful eye of the legal department. This is how it happened with Object Rexx, and this is how it happened with NetRexx. There is an important difference, in that NetRexx never was shipped as a product, except for a short time with z/VM, and there still under the ancient EWS licence.

NetRexx is now released under the ICU licence, which is a very liberal one, and RexxLA will take good care of it. The people who have been mobilized over the years are going to convene as the NetRexx language board, and development and releases will follow the great example that Mike Cowlishaw has set for his family of languages, which is talk it over and document first, seek agreement and only then implement.

The future of this great language is now safe, and there is no danger that it will languish on some corporate filesystem until it is inadvertently forgotten or erased. RexxLA will position itself to take on some more products as open source; but first we need to focus for some time on the future of NetRexx. Let us see if we can convene a language symposium to get some face-to-face time with the key players. This is only the beginning, and we will make sure that the world will hear more about NetRexx, the most pleasant computer language that yields very fast Java classes.

Rexx in MUSIC

Not the sort of MUSIC you can dance to, but nevertheless a small discovery, at least for me. On a trail that started from a YouTube clip on CTSS - a contemparary report by a young Fernando José "Corby" Corbato, leading to different Wikipedia articles on timesharing, I found MUSIC/SP amongst the TSS, MTS and TSO systems, a product of McGill University in Canada and once an IBM product - on of the timesharing systems for its mainframes. You can read about it at here. It is based on an earlier IBM timesharing system called RAX, of which hardly a trace is to be found on the net.

A working system is downloadable at here - it is runnable under a windows based simulator or under the multiplatform Hercules mainframe simulator. Being on a Mac, this is what I did. It offers a wealth in programming options and uses an interface that is quick to be grasped - including an ISPF/PDF like editor and panel system. You will need Hercules and a 3270TN Terminal Emulator. The system's console is a Telnet session, but the user terminals require the 3270 protocol. A Hercules configuration file is included, so you are good to go.

This whole menuing system is written in Rexx. When running CPS - the venerable clauses-per-second benchmark, I noticed that the system announces itself to "parse source" as CMS. Some inquiries later it became clear that the original VM Rexx interpreter code is part of MUSIC/SP. So everyone who has an interest in checking out the VM Interpreter can do so now using a legal download - most publicly available VM releases predate the release of Rexx and are a such of less interest to the Rexx community.

Rexx and other programming languages are runnable with a scheme that requires hardly any JCL - in fact none at all if you put the required magic incantion on the top of your script files. It is a nice environment for experimentation, and it is very laudable that these products have been preserved for posterity. A Rexx class, using the real interpreter, could be held in the cloud without any setup using IBM's (Rational now) 3270 on demand product which runs in the browser.

If you want to set up a MUSIC/SP system for yourself and you run into any trouble, just drop me a line.

2010 Rexx Symposium II

The second day, which was the first real symposium day - depending how you want to look at it - started with some challenges. Dave Ashley was not online, so we switched to Mark Hessling in Australia. Having ambushed Mark with webex (we originally wanted to use teamviewer, but were unsure about the options for sound - with me having some recent bad Skype experiences (before the outage) of call keeping ringing throughout the group call. Mark had no sound initially, and then it was very loud - and video looked initially dark and grim, but for his second presentation he had switched to his Mac and everything was fine. This first presentation was about the new functionality in Regina, the brilliant multiplatform Classic Rexx interpreter, and an advance peek into the THE editor in the GUI version.

Because the web conference was so new for all of us, we needed some time to fully adapt - switching off peoples mikes and seeing Kees Wiegel pop up with his video feed during various parts of the presentations.

After the break it was time for Rony who presented the 2010 edition of BSF4Rexx, the glue between Rexx and Java, followed by Jon Wolffers who saw his CSVStream class adopted into the newly released ooRexx 4.1.0. This led to some discussion about error handling and some late night development work by Jon - so it was a very fruitful presentation with changes being committed to the code base the same week.

Next was yours truly with my 'Building NetRexx Systems' presentation which was actually an expanded version of one not presented last year. This went sufficiently well but was hampered by the slides not refreshing very energetically over the webex. I could see this on the ipad which I used as webex monitor using a second userid and the webex ipad app. Probably my MacBook, already overtaxed by running Keynote, Lotus Notes and Sametime (work, you know) and hosting the webex conference, coupled with two high resolution screens, had to do with this. This cost us the live demo of the work done by Bill Finlason for the NetRexx plugin for Eclipse.

Rony did a presentation on using Rexx and ooRexx from NetRexx - its sibling as the title stated. This offered an interesting view on how the strong points of both environments can be combined. After this session we had a break and we continued, with a shrunken audience (especially locally) for the NetRexx architecture board. As web conferences go, compared to a face2face meeting, the medium hampered us somewhat in a free exchange of ideas, but left a strong impression that a conservative approach to the language proper is warranted, while work in Java integration is seen as desirable. I will do a separate writeup of this NLARB session, including the valuable email feedback I received afterwards.

The last symposium day was Tuesday, December 14, 2010. We started with Jaap Brugman, who presented on his work of coupling Rexx to cryptographic hardware for the Central Bank of the Netherlands, starting with a general introduction to cryptography and gradually delving into the intricacies of the C interface to the Rexx external functions that were used throughout his application.

Next was Robert John Wilson, who uses NetRexx to diagnose JDBC connections in his job as database specialist for IBM, His presentation was received very well, also because he demonstrated the tools that he uses live with real connections, and went through the sourcecode to illustrate the easy way in which this has been put together in NetRexx.

After the break Mark Hessling continued from Brisbane, with the earlier mentioned brilliant sound and video, and introduced two not yet released tools for Rexx, Rexx/CSV and Rexx/PDF. The former shows a different approach to CSV parsing that can be used from a Classic Rexx environment, and the latter nicely interfaces to a tool that produces Postscript and PDF.

Thomas Schneider from Vienna presented (from his study in Vienna) an overview of his product line, which has been put together using NetRexx exclusively, while his conversion products all convert to NetRexx.

Michiel van Hoorn was next, and delivered the last local (he lives in Almere) presentation of this 2010 Symposium, on his personal toolkit (aptly named 'Orde!' - which is the imperative for Order! in the Dutch language) which he employs for all his projects in his career as consultant and project manager.

The two last presentations, both from IBM'ers and both over the phone, were by David Ashley, who did a rundown of all changes and new functionality in ooRexx 4.1.0 as a repeat performance of his presentation in Winchester last year - and boy that ooRexx team has been busy the past year, and Harris Morgenstern, who added z/OS as a point of interest during this symposium again, with his very thorough presentation on System Rexx. This product is part of the z/OS simplification effort that is going on at IBM, and aims to make Rexx the perfect language to add system (console) commands to z/OS. I had to operate the slide pack during his talk, and what seemed to be an off-by-one error during the start of his talk, deteriorated with time. I had a copy of his slideset on my second screen, and it took some quick reading and adaption to show the slide corresponding with the words- I hope I did not do too bad a job.

Having the last presentations by one local person and for the rest remotely, enabled us to not lose half a day for travel - this might be an idea for subsequent symposia. So slightly past 18.00 hours local time we could conclude this 2010 - 21st - Rexx Language Symposium. It was interesting in the sense that it was a first for web access to the meeting, and this has led to a much larger attendance than the past few years. We also have learned a lot about how to conduct this, and this will certainly be usable for the next year. For the next time, we need sharper video, better sound systems (to feed a clip-on mike into the web conference) and a separate operator to remotely close mikes - and perhaps less quirky software, although it certainly was manageable.

So - I declare this symposium a moderate success and will proceed to prepare for the next one, probably in May 2011 on the other side of the atlantic ocean.

Rexx Language Symposium 2010

It felt bad not to have a symposium in 2010 - so we had one. We started with a dinner where Michael Dag, Rony Flatscher, Eugene Pesin and Michiel van Hoorn joined me and Venetia. It took place in the Ysbreker and was very nice; burgers and beer for most of us.

Sunday there was a set of tutorials planned. After leaving the web conference option dangerously untested for many days, we have settled on webex which seems to work well.. Rony is in the middle of his talk, and the webex was joined, at various times, by Bruce Skelly, K.P. Kirchdoerffer, Chuck mcKinnis, Gil Barmwater, Kees Wiegel, Walter Pachl, Mark Miesfeld, Tom Maynard, Thomas Schneider, David Requena, Donna di Meo, David Ashley, Pam Taylor, Chip Davis and Mike Cowlishaw already.

More later.